Inter-processor execution of configuration files on reconfigurable processors using smart network interface controller (smartnic) buffers

ABSTRACT

The technology disclosed relates to inter-processor execution of configuration files on reconfigurable processors using smart network interface controller (SmartNIC) buffers. In particular, the technology disclosed relates to a runtime logic that is configured to execute configuration files that define applications and process application data for applications using a first reconfigurable processor and a second reconfigurable processor. The execution includes streaming configuration data in the configuration files and the application data between the first reconfigurable processor and the second reconfigurable processor using one or more SmartNIC buffers.

PRIORITY APPLICATION

This application is a continuation of co-pending U.S. Non-provisionalpatent application Ser. No. 17/127,929, filed Dec. 18, 2020, entitled“INTER-NODE BUFFER-BASED STREAMING FOR RECONFIGURABLEPROCESSOR-AS-A-SERVICE (RPAAS)” (Attorney Docket No. SBNV 1029-2).

INCORPORATIONS

The following are incorporated by reference for all purposes as if fullyset forth herein:

U.S. Non-provisional patent application Ser. No. 17/127,929, filed Dec.18, 2020, entitled “INTER-NODE BUFFER-BASED STREAMING FOR RECONFIGURABLEPROCESSOR-AS-A-SERVICE (RPAAS)” (Attorney Docket No. SBNV 1029-2);

Prabhakar et al., “Plasticine: A Reconfigurable Architecture forParallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada;

Koeplinger et al., “Spatial: A Language And Compiler For ApplicationAccelerators,” Proceedings Of The 39th ACM SIGPLAN Conference OnProgramming Language Design And Implementation (PLDI), Proceedings ofthe 43rd International Symposium on Computer Architecture, 2018;

U.S. Non-provisional patent application Ser. No. 16/239,252, filed Jan.3, 2019, entitled, “VIRTUALIZATION OF A RECONFIGURABLE DATA PROCESSOR,”(Attorney Docket No. SBNV 1000-1);

U.S. Non-provisional patent application Ser. No. 16/197,826, filed Nov.21, 2018, entitled, “CONFIGURATION LOAD OF A RECONFIGURABLE DATAPROCESSOR,” (Attorney Docket No. SBNV 1001-1A);

U.S. Non-provisional patent application Ser. No. 16/198,086, filed Nov.21, 2018, entitled, “CONFIGURATION UNLOAD OF A RECONFIGURABLE DATAPROCESSOR,” (Attorney Docket No. SBNV 1001-1B);

U.S. Non-provisional patent application Ser. No. 16/260,548, filed Jan.29, 2019, entitled, “MATRIX NORMAL/TRANSPOSE READ AND A RECONFIGURABLEDATA PROCESSOR INCLUDING SAME,” (Attorney Docket No. SBNV 1005-1);

U.S. Non-provisional patent application Ser. No. 16/536,192, filed Aug.8, 2019, entitled, “COMPILER FLOW LOGIC FOR RECONFIGURABLEARCHITECTURES,” (Attorney Docket No. SBNV 1006-1);

U.S. Non-provisional patent application Ser. No. 16/407,675, filed May9, 2019, entitled, “CONTROL FLOW BARRIER AND RECONFIGURABLE DATAPROCESSOR,” (Attorney Docket No. SBNV 1007-1);

U.S. Non-provisional patent application Ser. No. 16/504,627, filed Jul.8, 2019, entitled, “QUIESCE RECONFIGURABLE DATA PROCESSOR,” (AttorneyDocket No. SBNV 1008-1);

U.S. Non-provisional patent application Ser. No. 16/572,516, filed Sep.16, 2019, entitled, “EFFICIENT EXECUTION OF OPERATION UNIT GRAPHS ONRECONFIGURABLE ARCHITECTURES BASED ON USER SPECIFICATION,” (AttorneyDocket No. SBNV 1009-2);

U.S. Non-provisional patent application Ser. No. 16/744,077, filed Jan.15, 2020, entitled, “COMPUTATIONALLY EFFICIENT SOFTMAX LOSS GRADIENTBACKPROPAGATION,” (Attorney Docket No. SBNV 1010-1);

U.S. Non-provisional patent application Ser. No. 16/590,058, filed Oct.1, 2019, entitled, “COMPUTATION UNITS FOR FUNCTIONS BASED ON LOOKUPTABLES,” (Attorney Docket No. SBNV 1011-1);

U.S. Non-provisional patent application Ser. No. 16/695,138, filed Nov.25, 2019, entitled, “COMPUTATION UNITS FOR BATCH NORMALIZATION,”(Attorney Docket No. SBNV 1012-1);

U.S. Non-provisional patent application Ser. No. 16/688,069, filed Nov.19, 2019, entitled, “LOOK-UP TABLE WITH INPUT OFFSETTING,” (AttorneyDocket No. SBNV 1013-1);

U.S. Non-provisional patent application Ser. No. 16/718,094, filed Dec.17, 2019, entitled, “COMPUTATION UNITS FOR ELEMENT APPROXIMATION,”(Attorney Docket No. SBNV 1014-1);

U.S. Non-provisional patent application Ser. No. 16/560,057, filed Sep.4, 2019, entitled, “SIGMOID FUNCTION IN HARDWARE AND A RECONFIGURABLEDATA PROCESSOR INCLUDING SAME,” (Attorney Docket No. SBNV 1015-1);

U.S. Non-provisional patent application Ser. No. 16/572,527, filed Sep.16, 2019, entitled, “PERFORMANCE ESTIMATION-BASED RESOURCE ALLOCATIONFOR RECONFIGURABLE ARCHITECTURES,” (Attorney Docket No. SBNV 1016-2);

U.S. Non-provisional patent application Ser. No. 15/930,381, filed May12, 2020, entitled, “COMPUTATIONALLY EFFICIENT GENERAL MATRIX-MATRIXMULTIPLICATION (GeMM),” (Attorney Docket No. SBNV 1019-1);

U.S. Non-provisional patent application Ser. No. 16/890,841, filed Jun.2, 2020, entitled, “ANTI-CONGESTION FLOW CONTROL FOR RECONFIGURABLEPROCESSORS,” (Attorney Docket No. SBNV 1021-1);

U.S. Non-provisional patent application Ser. No. 16/922,975, filed Jul.7, 2020, entitled, “RUNTIME VIRTUALIZATION OF RECONFIGURABLE DATAFLOWRESOURCES,” (Attorney Docket No. SBNV 1026-1);

US Non-provisional patent application Ser. No. 16/996,66, filed Aug. 18,2020, entitled, “RUNTIME PATCHING OF CONFIGURATION FILES,” (AttorneyDocket No. SBNV 1027-1);

U.S. Non-provisional patent application Ser. No. 17/023,015, filed Sep.16, 2020, “COMPILE TIME LOGIC FOR DETECTING STREAMING COMPATIBLE ANDBROADCAST COMPATIBLE DATA ACCESS PATTERNS” (Attorney Docket No. SBNV1022-1); and

U.S. Non-provisional patent application Ser. No. 17/031,679, filed Sep.24, 2020, “SYSTEMS AND METHODS FOR MEMORY LAYOUT DETERMINATION ANDCONFLICT RESOLUTION” (Attorney Docket No. SBNV 1023-1).

FIELD OF THE TECHNOLOGY DISCLOSED

The technology disclosed relates to latency optimization in intra-nodeand inter-node processing that uses processors like Central ProcessingUnits (CPUs), Graphics Processing Units (GPUs), Field Programmable GateArrays (FPGAs), Coarse-Grained Reconfigurable Architectures (CGRAs),Application-Specific Integrated Circuits (ASICs), Application SpecificInstruction-set Processor (ASIP), and Digital Signal Processors (DSPs).In particular, the technology disclosed relates to using buffers toefficiently stream data between processors on a same processing node andon different processing nodes.

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves can also correspond to implementations of the claimedtechnology.

Reconfigurable processors, including Field Programmable Gate Arrays(FPGAs), can be configured to implement a variety of functions moreefficiently or faster than might be achieved using a general-purposeprocessor executing a computer program. So-called Coarse-GrainedReconfigurable Architectures (CGRAs) are being developed in which theconfigurable units in the array are more complex than used in typical,more fine-grained FPGAs, and may enable faster or more efficientexecution of various classes of functions. For example, CGRAs have beenproposed that can enable implementation of energy-efficient acceleratorsfor machine learning and artificial intelligence workloads. See,Prabhakar, et al., “Plasticine: A Reconfigurable Architecture forParallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada.

Configuration of reconfigurable processors involves compilation of aconfiguration description to produce a configuration file, referred tosometimes as a bitstream or bit file, and distributing the configurationfile to the configurable units on the processor. To start a process, theconfiguration file must be loaded for that process. To change a process,the configuration file must be replaced with the new configuration file.

The procedures and supporting structures for distributing and loadingconfiguration files can be complex, and the execution of the procedurescan be time consuming.

In order to maximize operating efficiency and be able to run programs onmultiple reconfigurable processors on a same processing node ordifferent processing nodes, a means for efficiently streamingconfiguration data between reconfigurable processors is needed.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings, inwhich:

FIG. 1 shows an architectural level schematic of a data center inaccordance with an implementation.

FIG. 2A shows host sender buffers and host receiver buffers located in ahost memory of a first host processor of a first processing node in thedata center of FIG. 1.

FIG. 2B shows host sender buffers and host receiver buffers located in ahost memory of a second host processor of a second processing node inthe data center of FIG. 1.

FIG. 3A shows interface sender buffers and interface receiver bufferslocated at a first Network Interface Controller operatively coupled tothe first processing node.

FIG. 3B shows interface sender buffers and interface receiver bufferslocated at a second Network Interface Controller operatively coupled tothe second processing node.

FIG. 4A shows reconfigurable processor (RP) sender buffers andreconfigurable processor receiver buffers located in a processor memoryof a first reconfigurable processor operatively coupled to the firstprocessing node.

FIG. 4B shows reconfigurable processor sender buffers and reconfigurableprocessor receiver buffers located in a processor memory of a secondreconfigurable processor operatively coupled to the second processingnode.

FIG. 5A is a heuristics diagram of a runtime logic running at the firsthost processor.

FIG. 5B is a heuristics diagram of a runtime logic running at the secondhost processor.

FIG. 6 is a message sequence chart illustrating one implementation of adebugging logic running at the first host processor and detecting errorsin execution of configuration files on one or more of reconfigurableprocessors operatively coupled to the first processing node.

FIG. 7 is a message sequence chart illustrating one implementation ofthe debugging logic of FIG. 6 detecting errors in execution ofconfiguration files on one or more of reconfigurable processorsoperatively coupled to the second processing node.

FIG. 8 is a message sequence chart illustrating one implementation ofone or more of the reconfigurable processors operatively coupled to thefirst processing node issuing remote procedure calls to the first hostprocessor.

FIG. 9 is a message sequence chart illustrating one implementation ofone or more of the reconfigurable processors operatively coupled to thesecond processing node issuing remote procedure calls to the first hostprocessor.

FIG. 10 is a message sequence chart illustrating one implementation of atesting logic running at the first host processor and determining andreporting test statistics for execution of test configuration files onone or more of the reconfigurable processors operatively coupled to thefirst processing node.

FIG. 11 is a message sequence chart illustrating one implementation ofthe testing logic of FIG. 10 determining and reporting test statisticsfor execution of test configuration files on one or more of thereconfigurable processors operatively coupled to the second processingnode.

FIG. 12 is a message sequence chart illustrating one implementation ofexecuting a first set of functions in configuration files on one or moreof the reconfigurable processors operatively coupled to the firstprocessing node and executing a second set of functions in theconfiguration files on the first host processor.

FIG. 13 is a message sequence chart illustrating one implementation ofexecuting a first set of functions in configuration files on one or moreof the reconfigurable processors operatively coupled to the firstprocessing node and executing a second set of functions in theconfiguration files on the second host processor.

FIG. 14A shows sender and receiver buffers used by individualreconfigurable processors in the reconfigurable processors operativelycoupled to the first processing node for data streaming.

FIG. 14B shows sender and receiver buffers used by individualreconfigurable processors in the reconfigurable processors operativelycoupled to the second processing node for data streaming.

FIG. 15 is a message sequence chart illustrating one implementation ofexecuting a first set of functions in configuration files on a firstreconfigurable processor operatively coupled to the first processingnode and executing a second set of functions in the configuration fileson a second reconfigurable processor operatively coupled to the firstprocessing node.

FIG. 16 is a message sequence chart illustrating one implementation ofexecuting a first set of functions in configuration files on a firstreconfigurable processor operatively coupled to the first processingnode and executing a second set of functions in the configuration fileson a first reconfigurable processor operatively coupled to the secondprocessing node.

FIG. 17A is a message sequence chart illustrating one implementation ofasynchronous tensor streaming in which a next tensor is buffered while areconfigurable processor is processing a current tensor.

FIG. 17B is a message sequence chart illustrating one implementation ofasynchronous tensor streaming in which a next tensor is buffered beforea reconfigurable processor processes a current tensor.

FIG. 17C is a message sequence chart illustrating one implementation ofasynchronous tensor streaming in which a next tensor is buffered after areconfigurable processor has processed a current tensor.

FIG. 18 is a message sequence chart illustrating one implementation ofexecuting configuration files on reconfigurable processors that are ondifferent processing nodes in the data center.

FIG. 19 shows one implementation of memory mapping and allocatingvirtual buffers to physical buffers located in memories of differentnetwork components in the data center.

FIG. 20 shows an architectural level schematic of one implementation ofthe data center in which the processing nodes of the data center do notinclude host processors.

FIG. 21 is a message sequence chart illustrating one implementation ofbuffer-based inter-node streaming of configuration data over the networkfabric.

FIG. 22 is a message sequence chart illustrating another implementationof buffer-based inter-node streaming of configuration data over thenetwork fabric.

FIG. 23 illustrates one implementation of executing a model/applicationin parallel using the disclosed buffer-based inter-node streaming ofconfiguration data over the network fabric 136. This is referred toherein as “model parallelism.”

FIG. 24 illustrates one implementation of executing multiple instancesof a model/application in parallel using the disclosed buffer-basedinter-node streaming of configuration data over the network fabric 136.This is referred to herein as “data parallelism.”

FIG. 25 illustrates one implementation of executing configuration fileson heterogeneous reconfigurable processors.

FIG. 26 illustrates one implementation of executing configuration filesusing NIC or SmartNIC devices that are embedded on the reconfigurableprocessors.

FIG. 27 is a system diagram illustrating a system including a host, amemory, and an example reconfigurable data processor on which thetechnology disclosed can be applied.

FIG. 28 is a simplified block diagram of a top-level network andcomponents of a CGRA (Coarse-Grained Reconfigurable Architecture).

FIG. 29A is a simplified diagram of a tile and an array level networkusable in the configuration of FIG. 27, where the configurable units arenodes on the array level network and are configurable to implement aLook-Up Table with input offsetting.

FIG. 29B illustrates an example switch unit connecting elements in anarray level network.

FIG. 30 is a block diagram illustrating an example configurable unit,such as a Pattern Compute Unit (PCU).

FIG. 31 is a block diagram illustrating an example configurable unit,such as a Pattern Memory Unit (PMU).

DETAILED DESCRIPTION

The following discussion is presented to enable any person skilled inthe art to make and use the technology disclosed and is provided in thecontext of a particular application and its requirements. Variousmodifications to the disclosed implementations will be readily apparentto those skilled in the art, and the general principles defined hereinmay be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown but is to be accorded the widest scope consistentwith the principles and features disclosed herein.

Data Center

Systems and processes for providing ReconfigurableProcessor-as-a-Service (RPaaS) are described. The systems and processeswill be described with reference to FIG. 1 showing an architecturallevel schematic of a data center 100 in accordance with animplementation. Because FIG. 1 is an architectural diagram, certaindetails of the data center 100 are intentionally omitted to improve theclarity of the description. It may be noted that data center 100 caninclude the same, more, or fewer elements configured in the same ordifferent manner in other implementations. The discussion of FIG. 1 willbe organized as follows. First, the elements of the figure will bedescribed, followed by their interconnections. Then, the use of theelements in the system will be described in greater detail.

FIG. 1 shows first and second processing nodes in the data center 100.In FIG. 1, the first processing node is identified as “processing node1,” and the second processing node is identified as “processing node n.”The first and second processing nodes are configured to collaborativelyexecute configuration files for applications in a distributed fashion.One skilled in the art will appreciate that the data center 100 can haveany number of processing nodes operatively coupled for datacommunications through a network 136 (also called herein “network fabric136”). Examples of the network 136 include a Storage Area Network (SAN)and a Local Area Network (LAN). The SAN can be implemented with avariety of data communications fabrics, devices, and protocols. Forexample, the fabrics for the SAN can include Fibre Channel, Ethernet,InfiniBand, Serial Attached Small Computer System Interface (‘SAS’), orthe like. Data communications protocols for use with the SAN can includeAdvanced Technology Attachment (‘ATA’), Fibre Channel Protocol, SmallComputer System Interface (‘SCSI’), Internet Small Computer SystemInterface (‘iSCSI’), HyperSCSI, Non-Volatile Memory Express (‘NVMe’)over Fabrics, or the like.

The LAN can also be implemented with a variety of fabrics, devices, andprotocols. For example, the fabrics for the LAN can include Ethernet(802.3), wireless (802.11), or the like. Data communication protocolsfor use in the LAN can include Transmission Control Protocol (TCP′),User Datagram Protocol (‘UDP’), Internet Protocol (IP), HypertextTransfer Protocol (‘HTTP’), Wireless Access Protocol (‘WAP’), HandheldDevice Transport Protocol (‘HDTP’), Session Initiation Protocol (‘SIP’),Real-time Transport Protocol (‘RTP’), or the like.

The network 136 also connects other network components in the datacenter 100. Examples of other network components include buses,switches, routers, load balancers, hypervisors, and ApplicationProgramming Interfaces (APIs). Along the network 136, the switches, forexample, can receive packets via a plurality of input ports and cantransmit packets via a plurality of output ports. The processing nodesin the data center 100 can communicate with each other through thenetwork 136 using a variety of networking paths established by theswitches. Another example of the network 136 is a Wide Area Network(WAN).

A processing node (or node) is an addressable application running on ahardware device or virtual device that attaches to a network, and iscapable of sending, receiving, or forwarding information over acommunications channel to or from other processing nodes. Examples ofelectronic devices which can be deployed as hardware processing nodesinclude all varieties of computers, workstations, laptop computers,handheld computers, and smartphones. Processing nodes can be implementedin a cloud-based server system. More than one virtual device configuredas a processing node can be implemented using a single physical device.

The data center 100 comprises a pool of reconfigurable dataflowresources. The pool of reconfigurable dataflow resources can have avariety of compute scales and hierarchies. The pool of reconfigurabledataflow resources can be a single processing node operatively coupledto a plurality of reconfigurable processors, which in turn is supportedby different bus and memory resources. The processing node can have ahost processor (e.g., a CPU) that exchanges data with the reconfigurableprocessors, for example, over a local bus like Peripheral ComponentInterconnect Express (PCIe) interface. The host processor can have aruntime processor (or a runtime logic) that manages resource allocation,memory mapping, and execution of configuration files for applicationsrequesting execution from the host processor.

The pool of reconfigurable dataflow resources can be a rack (or cluster)of processing nodes connected through the network 136. Each processingnode in the rack can run a respective plurality of reconfigurableprocessors and include a respective host processor configured with arespective runtime processor. The runtime processors, distributed acrossthe processing nodes, communicate with each other to provide unifiedaccess to reconfigurable processors attached not only to their ownprocessing node but also to reconfigurable processors attached to everyother processing node in the data center 100.

The pool of reconfigurable dataflow resources can be a pod thatcomprises a plurality of racks connected through the network 136. Thepool of reconfigurable dataflow resources can be a superpod thatcomprises a plurality of pods connected through the network 136. Thepool of reconfigurable dataflow resources can be a zone that comprises aplurality of superpods connected through the network 136. The pool ofreconfigurable dataflow resources can be the data center 100 thatcomprises a plurality of zones connected through the network 136.

The pool of reconfigurable dataflow resources can include bus (ortransfer) resources. Examples of the bus resources include PCIechannels, Direct Memory Access (DMA) channels, and Double Data Rate(DDR) channels. The pool of reconfigurable dataflow resources caninclude memory (or storage) resources. Examples of the memory resourcesinclude main memory (e.g., off-chip/external Dynamic Random AccessMemory (DRAM), NAND flash), local secondary storage (e.g., local disks(e.g., HDD, SSD)), and remote secondary storage (e.g., distributed filesystems, web servers). Other examples of the memory resources includelatches, registers, flops, bypass networks, and caches (e.g., onesexplicitly addressed by RAMs/DRAMs/SRAMs). The pool of reconfigurabledataflow resources is dynamically scalable to meet the performancerequirements of applications requesting execution. The applicationsaccess the pool of reconfigurable dataflow resources over one or morenetworks (e.g., the Internet).

The discussion now returns to the first and second processing nodes ofthe data center 100. The first processing node comprises a first hostprocessor 102 a. Examples of the first host processor 102 a include x86and x64 processors. The first host processor 102 a interfaces with ahost memory 134 a (e.g., RAM). The first host processor 102 a has acompiler 112 a to compile applications and a runtime logic 122 a toexecute the compiled applications on a plurality of reconfigurableprocessors 142 a. The runtime logic 122 a is configured to provideon-demand access to the pool of reconfigurable dataflow resources, whichcan be rapidly provisioned and released with minimal management effortor service provider interaction.

Examples of the reconfigurable processors 142 a include FieldProgrammable Gate Arrays (FPGAs), Coarse-Grained ReconfigurableArchitectures (CGRAs), Application-Specific Integrated Circuits (ASICs),and Application Specific Instruction-set Processor (ASIP). Thereconfigurable processors 142 a interface with a reconfigurableprocessor memory 162 a (e.g., DRAM). Each of the reconfigurableprocessors 142 a includes an array of configurable units (e.g., computeunits and memory units) in a programmable interconnect fabric. The arrayof configurable units in a reconfigurable processor is partitionableinto a plurality of subarrays (or tiles) of configurable units.Additional details about one implementation of the architecture of thereconfigurable processors are discussed later in this application. Inother implementations, the processing nodes in the data center 100include processors instead of/in addition to the reconfigurableprocessors 142 a. Examples of such processors include GraphicsProcessing Units (GPUs) and Digital Signal Processors (DSPs).

A Network Interface Controller 132 a (e.g., NIC, SmartNIC) connects thefirst host processor 102 a and the reconfigurable processors 142 a tothe network 136. A bus switch 124 a uses local buses 125 a, 126 a, and127 a to operatively couple the first host processor 102 a, thereconfigurable processors 142 a, and the Network Interface Controller132 a. Examples of the local buses 125 a, 126 a, and 127 a includePeripheral Component Interconnect Express (PCIe), Cache CoherentInterconnect for Accelerators (CCIX), Compute Express Link (CXL), andOpen Coherent Accelerator Processor Interface (OpenCAPI).

The second processing node comprises a second host processor 102 n.Examples of the second host processor 102 n include x86 and x64processors. The second host processor 102 n interfaces with a hostmemory 134 n (e.g., RAM). The second host processor 102 n has a compiler112 n to compile applications and a runtime logic 122 n to execute thecompiled applications on a plurality of reconfigurable processors 142 n.The runtime logic 122 n is configured to provide on-demand access to thepool of reconfigurable dataflow resources, which can be rapidlyprovisioned and released with minimal management effort or serviceprovider interaction.

Examples of the reconfigurable processors 142 n include FieldProgrammable Gate Arrays (FPGAs), Coarse-Grained ReconfigurableArchitectures (CGRAs), Application-Specific Integrated Circuits (ASICs),and Application Specific Instruction-set Processor (ASIP). Thereconfigurable processors 142 n interface with a reconfigurableprocessor memory 162 n (e.g., DRAM). Each of the reconfigurableprocessors 142 n includes an array of configurable units (e.g., computeunits and memory units) in a programmable interconnect fabric. The arrayof configurable units in a reconfigurable processor is partitionableinto a plurality of subarrays (or tiles) of configurable units.Additional details about one implementation of the architecture of thereconfigurable processors are discussed later in this application. Inother implementations, the processing nodes in the data center 100include processors instead of/in addition to the reconfigurableprocessors 142 n. Examples of such processors include GraphicsProcessing Units (GPUs) and Digital Signal Processors (DSPs).

A Network Interface Controller 132 n (e.g., NIC, SmartNIC) connects thesecond host processor 102 n and the reconfigurable processors 142 n tothe network 136. A bus switch 124 n uses local buses 125 n, 126 n, and127 n to operatively couple the second host processor 102 n, thereconfigurable processors 142 n, and the Network Interface Controller132 n. Examples of the local buses 125 n, 126 n, and 127 n includePeripheral Component Interconnect Express (PCIe), Cache CoherentInterconnect for Accelerators (CCIX), Compute Express Link (CXL), andOpen Coherent Accelerator Processor Interface (OpenCAPI).

Having described the elements and interconnections of FIG. 1, thediscussion now turns to the buffers used by the technology disclosed forlatency optimization in intra-node and inter-node processing.

Buffers

FIG. 2A shows host sender buffers 212 a and host receiver buffers 202 alocated in the host memory 134 a. The host sender buffers 212 a arereconfigurable processors-to-host processor buffers that are configuredto receive data from the reconfigurable processors 142 a and provide thedata to the first host processor 102 a. The host receiver buffers 202 aare host processor-to-reconfigurable processors buffers that areconfigured to receive data from the first host processor 102 a andprovide the data to the reconfigurable processors 142 a. Examples of thedata include scalar data (e.g., control bits) and vector data (e.g.,vectors, tensors, arguments, commands). The host memory 134 a, andtherefore the host sender buffers 212 a and the host receiver buffers202 a, are accessible to each of the host processors (e.g., first andsecond host processors 102 a, 102 n), each of the reconfigurableprocessors (e.g., reconfigurable processors 142 a, 142 n), and each ofthe Network Interface Controllers (e.g., Network Interface Controllers132 a, 132 n) in the data center 100. The host sender buffers 212 a andthe host receiver buffers 202 a can be First-In, First-Out (FIFO)buffers, First-In, Last-Out (FILO) buffers, Last-In, First-Out (LIFO)buffers, Last-In, Last-Out (LILO) buffers, or circular buffers. The hostsender buffers 212 a and the host receiver buffers 202 a can be of size8 bytes, 16 bytes, 32 bytes, 64 bytes, 128 bytes, 256 bytes, and so on,or any convenient size appropriate for the transfer of data between thehost processor, the network interface controllers, and thereconfigurable processors.

FIG. 2B shows host sender buffers 212 n and host receiver buffers 202 nlocated in the host memory 134 n. The host sender buffers 212 n arereconfigurable processors-to-host processor buffers that are configuredto receive data from the reconfigurable processors 142 n and provide thedata to the second host processor 102 n. The host receiver buffers 202 nare host processor-to-reconfigurable processors buffers that areconfigured to receive data from the second host processor 102 n andprovide the data to the reconfigurable processors 142 n. Examples of thedata include scalar data (e.g., control bits) and vector data (e.g.,vectors, tensors, arguments, commands). The host memory 134 n, andtherefore the host sender buffers 212 n and the host receiver buffers202 n, are accessible to each of the host processors (e.g., first andsecond host processors 102 a, 102 n), each of the reconfigurableprocessors (e.g., reconfigurable processors 142 a, 142 n), and each ofthe Network Interface Controllers (e.g., Network Interface Controllers132 a, 132 n) in the data center 100. The host sender buffers 212 n andthe host receiver buffers 202 n can be First-In, First-Out (FIFO)buffers, First-In, Last-Out (FILO) buffers, Last-In, First-Out (LIFO)buffers, Last-In, Last-Out (LILO) buffers, or circular buffers. The hostsender buffers 212 n and the host receiver buffers 202 n can be of size8 bytes, 16 bytes, 32 bytes, 64 bytes, 128 bytes, 256 bytes, and so on,or any convenient size appropriate for the transfer of data between thehost processor, the network interface controllers, and thereconfigurable processors.

FIG. 3A shows interface sender buffers 312 a and interface receiverbuffers 302 a located at the Network Interface Controller 132 a. Theinterface sender buffers 312 a are reconfigurable processors-to-hostprocessor buffers that are configured to receive data from thereconfigurable processors 142 a and provide the data to the first hostprocessor 102 a. The interface receiver buffers 302 a are hostprocessor-to-reconfigurable processors buffers that are configured toreceive data from the first host processor 102 a and provide the data tothe reconfigurable processors 142 a. Examples of the data include scalardata (e.g., control bits) and vector data (e.g., vectors, tensors,arguments, commands). The Network Interface Controller 132 a, andtherefore the interface sender buffers 312 a and the interface receiverbuffers 302 a, are accessible to each of the host processors (e.g.,first and second host processors 102 a, 102 n), each of thereconfigurable processors (e.g., reconfigurable processors 142 a, 142n), and each of the Network Interface Controllers (e.g., NetworkInterface Controllers 132 a, 132 n) in the data center 100. Theinterface sender buffers 312 a and the interface receiver buffers 302 acan be First-In, First-Out (FIFO) buffers, First-In, Last-Out (FILO)buffers, Last-In, First-Out (LIFO) buffers, Last-In, Last-Out (LILO)buffers, or circular buffers. The interface sender buffers 312 a and theinterface receiver buffers 302 a can be of size 8 bytes, 16 bytes, 32bytes, 64 bytes, 128 bytes, 256 bytes, and so on, or any convenient sizeappropriate for the transfer of data between the host processor, thenetwork interface controllers, and the reconfigurable processors.

FIG. 3B shows interface sender buffers 312 n and interface receiverbuffers 302 n located at the Network Interface Controller 132 n. Theinterface sender buffers 312 n are reconfigurable processors-to-hostprocessor buffers that are configured to receive data from thereconfigurable processors 142 n and provide the data to the second hostprocessor 102 n. The interface receiver buffers 302 n are hostprocessor-to-reconfigurable processors buffers that are configured toreceive data from the second host processor 102 n and provide the datato the reconfigurable processors 142 n. Examples of the data includescalar data (e.g., control bits) and vector data (e.g., vectors,tensors, arguments, commands). The Network Interface Controller 132 n,and therefore the interface sender buffers 312 n and the interfacereceiver buffers 302 n, are accessible to each of the host processors(e.g., first and second host processors 102 a, 102 n), each of thereconfigurable processors (e.g., reconfigurable processors 142 a, 142n), and each of the Network Interface Controllers (e.g., NetworkInterface Controllers 132 a, 132 n) in the data center 100. Theinterface sender buffers 312 n and the interface receiver buffers 302 ncan be First-In, First-Out (FIFO) buffers, First-In, Last-Out (FILO)buffers, Last-In, First-Out (LIFO) buffers, Last-In, Last-Out (LILO)buffers, or circular buffers. The interface sender buffers 312 n and theinterface receiver buffers 302 n can be of size 8 bytes, 16 bytes, 32bytes, 64 bytes, 128 bytes, 256 bytes, and so on, or any convenient sizeappropriate for the transfer of data between the host processor, thenetwork interface controllers, and the reconfigurable processors.

FIG. 4A shows reconfigurable processor (RP) sender buffers 412 a andreconfigurable processor (RP) receiver buffers 402 a located in thereconfigurable processor memory 162 a of the reconfigurable processors142 a. The reconfigurable processor sender buffers 412 a arereconfigurable processors-to-host processor buffers that are configuredto receive data from the reconfigurable processors 142 a and provide thedata to the first host processor 102 a. The reconfigurable processorreceiver buffers 402 a are host processor-to-reconfigurable processorsbuffers that are configured to receive data from the first hostprocessor 102 a and provide the data to the reconfigurable processors142 a. Examples of the data include scalar data (e.g., control bits) andvector data (e.g., vectors, tensors, arguments, commands). Thereconfigurable processor memory 162 a, and therefore the reconfigurableprocessor sender buffers 412 a and the reconfigurable processor receiverbuffers 402 a, are accessible to each of the host processors (e.g.,first and second host processors 102 a, 102 n), each of thereconfigurable processors (e.g., reconfigurable processors 142 a, 142n), and each of the Network Interface Controllers (e.g., NetworkInterface Controllers 132 a, 132 n) in the data center 100. Thereconfigurable processor sender buffers 412 a and the reconfigurableprocessor receiver buffers 402 a can be First-In, First-Out (FIFO)buffers, First-In, Last-Out (FILO) buffers, Last-In, First-Out (LIFO)buffers, Last-In, Last-Out (LILO) buffers, or circular buffers. Thereconfigurable processor sender buffers 412 a and the reconfigurableprocessor receiver buffers 402 a can be of size 8 bytes, 16 bytes, 32bytes, 64 bytes, 128 bytes, 256 bytes, and so on, or any convenient sizeappropriate for the transfer of data between the host processor, thenetwork interface controllers, and the reconfigurable processors.

FIG. 4B shows reconfigurable processor (RP) sender buffers 412 n andreconfigurable processor (RP) receiver buffers 402 n located in thereconfigurable processor memory 162 n of the reconfigurable processors142 n. The reconfigurable processor sender buffers 412 n arereconfigurable processors-to-host processor buffers that are configuredto receive data from the reconfigurable processors 142 n and provide thedata to the second host processor 102 n. The reconfigurable processorreceiver buffers 402 n are host processor-to-reconfigurable processorsbuffers that are configured to receive data from the second hostprocessor 102 n and provide the data to the reconfigurable processors142 n. Examples of the data include scalar data (e.g., control bits) andvector data (e.g., vectors, tensors, arguments, commands). Thereconfigurable processor memory 162 n, and therefore the reconfigurableprocessor sender buffers 412 n and the reconfigurable processor receiverbuffers 402 n, are accessible to each of the host processors (e.g.,first and second host processors 102 a, 102 n), each of thereconfigurable processors (e.g., reconfigurable processors 142 a, 142n), and each of the Network Interface Controllers (e.g., NetworkInterface Controllers 132 a, 132 n) in the data center 100. Thereconfigurable processor sender buffers 412 n and the reconfigurableprocessor receiver buffers 402 n can be First-In, First-Out (FIFO)buffers, First-In, Last-Out (FILO) buffers, Last-In, First-Out (LIFO)buffers, Last-In, Last-Out (LILO) buffers, or circular buffers. Thereconfigurable processor sender buffers 412 n and the reconfigurableprocessor receiver buffers 402 n can be of size 8 bytes, 16 bytes, 32bytes, 64 bytes, 128 bytes, 256 bytes, and so on, or any convenient sizeappropriate for the transfer of data between the host processor, thenetwork interface controllers, and the reconfigurable processors.

The buffers can be defined by a virtual address space that maps to aphysical range of memory addresses (which may be contiguous ordiscontiguous) in the memory. The virtual buffers are read from andwritten to at locations in the memory indicated using a read pointer andwrite pointer, respectfully. The pointers are held in a memory (whichmay be the same as or separate to memory).

Having described the buffers, the discussion now turns to the debugginglogic and the testing logic disclosed by the technology disclosed.

Debugging Logic and Testing Logic

FIG. 5A is a heuristics diagram of the runtime logic 122 a. The runtimelogic 122 a comprises debugging logic 502 a and testing logic 512 a. Theruntime logic 122 a is configured to load and execute one or moreconfiguration files for applications on one or more of thereconfigurable processors 142 a. The reconfigurable processors 142 a areconfigured to process the configuration files and generate outputs, andto send the outputs to the first host processor 102 a using at least oneof the reconfigurable processors-to-host processor buffers (e.g., hostsender buffers 212 a, host sender buffers 212 n, interface senderbuffers 312 a, interface sender buffers 312 n, reconfigurable processorsender buffers 412 a, reconfigurable processor sender buffers 412 n).

The debugging logic 502 a, running on the first host processor 102 a, isconfigured to detect errors (e.g., in execution of the configurationfiles). In one implementation, the debugging logic 502 a is furtherconfigured to report the errors to a debugging console on the first hostprocessor 102 a based on comparison of the outputs to expected outputs.In another implementation, the debugging logic 502 a is furtherconfigured to report the errors to a debug output file on the first hostprocessor 102 a based on the comparison of the outputs to the expectedoutputs.

In some implementations, debugging logic running on a particular hostprocessor or reconfigurable processor in the data center 100 can reporterrors to any other host processor or reconfigurable processor in thedata center 100. For example, the debugging logic 502 a, running on thefirst host processor 102 a, can report errors to a debugging console onthe second host processor 102 n based on comparison of outputs toexpected outputs. In another example, the debugging logic 502 a canreport errors to a debug output file on the second host processor 102 nbased on comparison of outputs to expected outputs.

The runtime logic 122 a is further configured to execute, on thereconfigurable processors 142 a, one or more test configuration filesfor test applications. The reconfigurable processors 142 a are furtherconfigured to process the test configuration files and generate testoutputs, and to send the test outputs to the first host processor 102 ausing at least one of the reconfigurable processors-to-host processorbuffers (e.g., host sender buffers 212 a, host sender buffers 212 n,interface sender buffers 312 a, interface sender buffers 312 n,reconfigurable processor sender buffers 412 a, reconfigurable processorsender buffers 412 n). The testing logic 512 a, running on the firsthost processor 102 a, is configured to determine test statistics basedon the test outputs, and to report the test statistics to a test outputfile on the first host processor 102 a.

In some implementations, testing logic running on a particular hostprocessor or reconfigurable processor in the data center 100 can reporttest statistics to a test output file on any other host processor orreconfigurable processor in the data center 100. For example, thetesting logic 512 a, running on the first host processor 102 a, canreport test statistics to a test output file on the second hostprocessor 102 n.

FIG. 5B is a heuristics diagram of the runtime logic 122 n. The runtimelogic 122 n comprises debugging logic 502 n and testing logic 512 n. Theruntime logic 122 n is configured to load and execute one or moreconfiguration files for applications on one or more of thereconfigurable processors 142 n. The reconfigurable processors 142 n areconfigured to process the configuration files and generate outputs, andto send the outputs to the second host processor 102 n using at leastone of the reconfigurable processors-to-host processor buffers (e.g.,host sender buffers 212 a, host sender buffers 212 n, interface senderbuffers 312 a, interface sender buffers 312 n, reconfigurable processorsender buffers 412 a, reconfigurable processor sender buffers 412 n).

The debugging logic 502 n, running on the second host processor 102 n,is configured to detect errors (e.g., in execution of the configurationfiles). In one implementation, the debugging logic 502 n is furtherconfigured to report errors to a debugging console on the second hostprocessor 102 n based on comparison of the outputs to expected outputs.In another implementation, the debugging logic 502 n is furtherconfigured to report the errors to a debug output file on the secondhost processor 102 n based on the comparison of the outputs to theexpected outputs.

In some implementations, debugging logic running on a particular hostprocessor or reconfigurable processor in the data center 100 can reporterrors to any other host processor or reconfigurable processor in thedata center 100. For example, the debugging logic 502 n, running on thesecond host processor 102 n, can report errors to a debugging console onthe first host processor 102 a based on comparison of outputs toexpected outputs. In another example, the debugging logic 502 n canreport errors to a debug output file on the first host processor 102 abased on comparison of outputs to expected outputs.

In some implementations, testing logic running on a particular hostprocessor or reconfigurable processor in the data center 100 can reporttest statistics to a test output file on any other host processor orreconfigurable processor in the data center 100. For example, thetesting logic 512 n, running on the second host processor 102 n, canreport test statistics to a test output file on the first host processor102 a.

FIG. 6 is a message sequence chart 600 illustrating one implementationof the debugging logic 502 a detecting errors in execution ofconfiguration files on one or more of the reconfigurable processors (RP)142 a. At operation one, the compiler 112 a compiles an application 602to generate a graph that includes one or more configuration files forthe application 602. At operation two, the compiler 112 a sends thegraph to the runtime logic 122 a for execution. At operation three, theruntime logic 122 a loads and executes the configuration files on one ormore of the reconfigurable processors 142 a. At operation four, thereconfigurable processors 142 a process the configuration files andgenerate outputs (e.g., vectors, tensors). At operation five, thereconfigurable processors 142 a send the outputs to sender buffers 632(or reconfigurable processors-to-host processor buffers). Examples ofthe sender buffers 632 include host sender buffers 212 a, host senderbuffers 212 n, interface sender buffers 312 a, interface sender buffers312 n, reconfigurable processor sender buffers 412 a, and reconfigurableprocessor sender buffers 412 n. At operation six, the sender buffers 632provide the outputs to the debugging logic 502 a. At operation seven,the debugging logic 502 a detects errors in the execution of theconfiguration files based on comparison of the outputs to expectedoutputs. At operation eight, the debugging logic 502 a reports theerrors to a debugging console or a debug output file on the first hostprocessor 102 a. Other implementations may perform the operations indifferent orders and/or with different, fewer, or additional operationsthan the ones illustrated in FIG. 6. Multiple operations can be combinedin some implementations.

One skilled in the art will appreciate that, in FIG. 6, operations threeand six comprise streaming network packets between reconfigurableprocessors (e.g., RPs 142 a) and a host processor (e.g., host 102 a) ona same processing node 1 over local buses (e.g., PCIe buses) using aprotocol like Transmission Control Protocol (TCP).

FIG. 7 is a message sequence chart 700 illustrating one implementationof the debugging logic 502 a detecting errors in execution ofconfiguration files on one or more of the reconfigurable processors (RP)142 n. At operation one, the compiler 112 a compiles an application 702to generate a graph that includes one or more configuration files forthe application 702. At operation two, the compiler 112 a sends thegraph to the runtime logic 122 a for execution. At operation three, theruntime logic 122 a loads and executes the configuration files on one ormore of the reconfigurable processors 142 n. At operation four, thereconfigurable processors 142 n process the configuration files andgenerate outputs (e.g., vectors, tensors). At operation five, thereconfigurable processors 142 n send the outputs to sender buffers 732(or reconfigurable processors-to-host processor buffers). Examples ofthe sender buffers 732 include host sender buffers 212 a, host senderbuffers 212 n, interface sender buffers 312 a, interface sender buffers312 n, reconfigurable processor sender buffers 412 a, and reconfigurableprocessor sender buffers 412 n. At operation six, the sender buffers 732provide the outputs to the debugging logic 502 a. At operation seven,the debugging logic 502 a detects errors in the execution of theconfiguration files based on comparison of the outputs to expectedoutputs. At operation eight, the debugging logic 502 a reports theerrors to a debugging console or a debug output file on the first hostprocessor 102 a. Other implementations may perform the operations indifferent orders and/or with different, fewer, or additional operationsthan the ones illustrated in FIG. 7. Multiple operations can be combinedin some implementations.

One skilled in the art will appreciate that, in FIG. 7, operations threeand six comprise streaming network packets between one or morereconfigurable processors (e.g., RPs 142 n) on the second processingnode and a host processor (e.g., host 102 a) on the first processingnode over the network fabric 136 (e.g., Ethernet, InfiniBand (IB)) usingprotocols like RDMA over Converged Ethernet (RoCE), TCP, User DatagramProtocol (UDP), and Quick UDP Internet Connections (QUIC).

FIG. 8 is a message sequence chart 800 illustrating one implementationof one or more of the reconfigurable processors (RP) 142 a issuingremote procedure calls to the first host processor 102 a. At operationone, the compiler 112 a compiles an application 802 to generate a graphthat includes one or more configuration files for the application 802.At operation two, the compiler 112 a sends the graph to the runtimelogic 122 a for execution. At operation three, the runtime logic 122 aloads and executes the configuration files on one or more of thereconfigurable processors 142 a. At operation four, the reconfigurableprocessors 142 a process the configuration files and generate outputs(e.g., vectors, tensors). At operation five, the reconfigurableprocessors 142 a issue one or more remote procedure calls to the firsthost processor 102 a using sender buffers 832 (or reconfigurableprocessors-to-host processor buffers). Examples of the sender buffers832 include host sender buffers 212 a, host sender buffers 212 n,interface sender buffers 312 a, interface sender buffers 312 n,reconfigurable processor sender buffers 412 a, and reconfigurableprocessor sender buffers 412 n. In one implementation, thereconfigurable processors 142 a notify the first host processor 102 a oferror reporting using the remote procedure calls. At operation six, thereconfigurable processors 142 a use at least one of the sender buffers832 to send one or more argument values to the first host processor 102a for execution of the remote procedure calls. At operation seven, thesender buffers 832 provide the remote procedure calls and the argumentvalues to the runtime logic 122 a. At operation 8, one or more responsesto the remote procedure calls are sent to the reconfigurable processors142 n via the buffers (e.g., sender buffers of the first host processor102 a and receiver buffers of the reconfigurable processors 142 a).Other implementations may perform the operations in different ordersand/or with different, fewer, or additional operations than the onesillustrated in FIG. 8. Multiple operations can be combined in someimplementations.

One skilled in the art will appreciate that, in FIG. 8, operations threeand seven comprise streaming network packets between reconfigurableprocessors (e.g., RPs 142 a) and a host processor (e.g., host 102 a) ona same processing node 1 over local buses (e.g., PCIe buses) using aprotocol like Transmission Control Protocol (TCP).

FIG. 9 is a message sequence chart 900 illustrating one implementationof one or more of the reconfigurable processors (RP) 142 n issuingremote procedure calls to the first host processor 102 a. At operationone, the compiler 112 a compiles an application 902 to generate a graphthat includes one or more configuration files for the application 902.At operation two, the compiler 112 a sends the graph to the runtimelogic 122 a for execution. At operation three, the runtime logic 122 aloads and executes the configuration files on one or more of thereconfigurable processors 142 n. At operation four, the reconfigurableprocessors 142 n process the configuration files and generate outputs(e.g., vectors, tensors). At operation five, the reconfigurableprocessors 142 n issue one or more remote procedure calls to the firsthost processor 102 a using sender buffers 932 (or reconfigurableprocessors-to-host processor buffers). Examples of the sender buffers932 include host sender buffers 212 a, host sender buffers 212 n,interface sender buffers 312 a, interface sender buffers 312 n,reconfigurable processor sender buffers 412 a, and reconfigurableprocessor sender buffers 412 n. In one implementation, thereconfigurable processors 142 n notify the first host processor 102 a oferror reporting using the remote procedure calls. At operation six, thereconfigurable processors 142 n use at least one of the sender buffers932 to send one or more argument values to the first host processor 102a for execution of the remote procedure calls. At operation seven, thesender buffers 932 provide the remote procedure calls and the argumentvalues to the runtime logic 122 a. At operation 8, one or more responsesto the remote procedure calls are sent to the reconfigurable processors142 n via the buffers (e.g., sender buffers of the first host processor102 a and receiver buffers of the reconfigurable processors 142 n).Other implementations may perform the operations in different ordersand/or with different, fewer, or additional operations than the onesillustrated in FIG. 9. Multiple operations can be combined in someimplementations.

One skilled in the art will appreciate that, in FIG. 9, operations threeand seven comprise streaming network packets between one or morereconfigurable processors (e.g., RPs 142 n) on the second processingnode and a host processor (e.g., host 102 a) on the first processingnode over the network fabric 136 (e.g., Ethernet, InfiniBand (IB)) usingprotocols like RDMA over Converged Ethernet (RoCE), TCP, User DatagramProtocol (UDP), and Quick UDP Internet Connections (QUIC).

FIG. 10 is a message sequence chart 1000 illustrating one implementationof the testing logic 512 a reporting test statistics for execution oftest configuration files on one or more of the reconfigurable processors(RP) 142 a. At operation one, the compiler 112 a compiles a testapplication 1002 to generate a test graph that includes one or more testconfiguration files for the test application 1002. At operation two, thecompiler 112 a sends the test graph to the runtime logic 122 a forexecution. At operation three, the runtime logic 122 a loads andexecutes the test configuration files on one or more of thereconfigurable processors 142 a. At operation four, the reconfigurableprocessors 142 a process the test configuration files and generate testoutputs (e.g., vectors, tensors). At operation five, the reconfigurableprocessors 142 a send the test outputs to sender buffers 1032 (orreconfigurable processors-to-host processor buffers). Examples of thesender buffers 1032 include host sender buffers 212 a, host senderbuffers 212 n, interface sender buffers 312 a, interface sender buffers312 n, reconfigurable processor sender buffers 412 a, and reconfigurableprocessor sender buffers 412 n. At operation six, the sender buffers1032 provide the test outputs to the testing logic 512 a. At operationseven, the testing logic 512 a determines test statistics based on thetest outputs. At operation eight, the testing logic 512 a reports thetest statistics to a test output file on the first host processor 102 a.Other implementations may perform the operations in different ordersand/or with different, fewer, or additional operations than the onesillustrated in FIG. 10. Multiple operations can be combined in someimplementations.

One skilled in the art will appreciate that, in FIG. 10, operationsthree and six comprise streaming network packets between reconfigurableprocessors (e.g., RPs 142 a) and a host processor (e.g., host 102 a) ona same processing node 1 over local buses (e.g., PCIe buses) using aprotocol like Transmission Control Protocol (TCP).

FIG. 11 is a message sequence chart 1100 illustrating one implementationof the testing logic 512 a reporting test statistics for execution oftest configuration files on one or more of the reconfigurable processors(RP) 142 n. At operation one, the compiler 112 a compiles a testapplication 1102 to generate a test graph that includes one or more testconfiguration files for the test application 1102. At operation two, thecompiler 112 a sends the test graph to the runtime logic 122 a forexecution. At operation three, the runtime logic 122 a loads andexecutes the test configuration files on one or more of thereconfigurable processors 142 n. At operation four, the reconfigurableprocessors 142 n process the test configuration files and generate testoutputs (e.g., vectors, tensors). At operation five, the reconfigurableprocessors 142 n send the test outputs to sender buffers 1132 (orreconfigurable processors-to-host processor buffers). Examples of thesender buffers 1132 include host sender buffers 212 a, host senderbuffers 212 n, interface sender buffers 312 a, interface sender buffers312 n, reconfigurable processor sender buffers 412 a, and reconfigurableprocessor sender buffers 412 n. At operation six, the sender buffers1132 provide the test outputs to the testing logic 512 a. At operationseven, the testing logic 512 a determines test statistics based on thetest outputs. At operation eight, the testing logic 512 a reports thetest statistics to a test output file on the first host processor 102 a.Other implementations may perform the operations in different ordersand/or with different, fewer, or additional operations than the onesillustrated in FIG. 11. Multiple operations can be combined in someimplementations.

One skilled in the art will appreciate that, in FIG. 11, operationsthree and six comprise streaming network packets between one or morereconfigurable processors (e.g., RPs 142 n) on the second processingnode and a host processor (e.g., host 102 a) on the first processingnode over the network fabric 136 (e.g., Ethernet, InfiniBand (IB)) usingprotocols like RDMA over Converged Ethernet (RoCE), TCP, User DatagramProtocol (UDP), and Quick UDP Internet Connections (QUIC).

Having described the debugging logic and the testing logic, thediscussion now turns to the reconfigurable processor-to-host processorworkload sharing disclosed by the technology disclosed.

Reconfigurable Processor-to-Host Processor Workload Sharing

FIG. 12 is a message sequence chart 1200 illustrating one implementationof executing a first set of functions in configuration files and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) on one or more of the reconfigurable processors (RP)142 a and executing a second set of functions and/or data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) in the configuration files on the first host processor 102 a.At operation one, the compiler 112 a receives an application 1202 forcompilation. At operation two, the compiler 112 a compiles theapplication 1202 to generate one or more configuration files 1212. Theconfiguration files 1212 include a plurality of functions. The pluralityof functions includes a first set of functions 1214 and a second set offunctions 1224. Examples of functions in the plurality of functionsinclude non-linearities like Rectified Linear Unit (ReLU) and itsvariants (e.g., leaky ReLU), hyperbolic tangent, sigmoid, and softmax,element-wise addition, matrix multiplication (e.g., General MatrixMultiply (GeMM)), layer normalization (e.g., batch normalization), lossfunctions like cross-entropy, and tensor shape modifiers like transpose.At operation three, the compiler 112 a sends the configuration files1212 to the runtime logic 122 a for execution. At operation four, theruntime logic 122 a loads the first set of functions 1214 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) and the second set of functions 1224 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) on one or more of the reconfigurable processors142 a. At operation five, the reconfigurable processors 142 a processthe first set of functions 1214 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) and generatea first set of outputs (e.g., vectors, tensors). The reconfigurableprocessors 142 a transmit functions in the second set of functions 1224and/or the data therefor (e.g., weights, coefficients, vectors, tensors(image data, audio data, natural language processing (NLP data), controldata (e.g., control tokens)) to the first host processor 102 a using oneor more reconfigurable processors-to-host processor buffers. This isreferred to herein as “reconfigurable processor-to-host processorworkload sharing.” In one implementation, data on which the functions inthe second set of functions 1224 are executed is transmitted to thefirst host processor 102 a using the reconfigurable processors-to-hostprocessor buffers. In some implementations, respective ones of thereconfigurable processors-to-host processor buffers are used to transmitrespective functions in the second set of functions 1224 and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) to the first host processor 102 a. One example workloadsharing flow includes using one or more of the reconfigurable processorsender buffers 412 a and one or more of the host receiver buffers 202 a.At operation six, the reconfigurable processors 142 a transmit thefunctions in the second set of functions 1224 and/or the data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) to the reconfigurable processor sender buffers 412 a. Atoperation seven, the reconfigurable processor sender buffers 412 atransmit the functions in the second set of functions 1224 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) to the host receiver buffers 202 a. At operationeight, the host receiver buffers 202 a transmit the functions in thesecond set of functions 1224 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) to the firsthost processor 102 a. At operation nine, the first host processor 102 aexecutes the functions in the second set of functions 1224 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) to generate a second set of outputs (or results1234) (e.g., vectors, tensors). The first host processor 102 a transmitsthe results 1234 to one or more of the reconfigurable processors 142 ausing one or more host processor-to-reconfigurable processors buffers.In some implementations, respective ones of the hostprocessor-to-reconfigurable processors buffers are used to transmitrespective results of executing respective functions in the second setof functions 1224 and/or data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) to the reconfigurableprocessors 142 a. One workload sharing flow includes using one or moreof the host sender buffers 212 a and one or more of the reconfigurableprocessor receiver buffers 402 a. At operation ten, the first hostprocessor 102 a transmits the results 1234 to the host sender buffers212 a. At operation eleven, the host sender buffers 212 a transmit theresults 1234 to the reconfigurable processor receiver buffers 402 a. Atoperation twelve, the reconfigurable processor receiver buffers 402 atransmit the results 1234 to the reconfigurable processors 142 a. Insome implementations, one or more functions in the first set offunctions 1214 waits for results of execution of one or more functionsin the second set of functions 1224 and/or the data therefor (e.g.,weights, coefficients, vectors, tensors (image data, audio data, naturallanguage processing (NLP data), control data (e.g., control tokens)) onthe first host processor 102 a to combine the results with results ofexecution of one or more functions in the first set of functions 1214and/or the data therefor (e.g., weights, coefficients, vectors, tensors(image data, audio data, natural language processing (NLP data), controldata (e.g., control tokens)) on the reconfigurable processors 142 a. Inother implementations, the first set of functions 1214 and the secondset of functions 1224 operate separately and in parallel. In oneimplementation, one or more functions in the second set of functions1224 daisy chains the results to one or more functions in the first setof functions 1214, and vice-versa. In another implementation, one ormore functions in the second set of functions 1224 executes for acertain number of iterations before returning the results to thereconfigurable processors 142 a. Other implementations may perform theoperations in different orders and/or with different, fewer, oradditional operations than the ones illustrated in FIG. 12. Multipleoperations can be combined in some implementations.

One skilled in the art will appreciate that, in FIG. 12, operations six,seven, eight, ten, eleven, and twelve comprise streaming network packetsbetween reconfigurable processors (e.g., RPs 142 a) and a host processor(e.g., host 102 a) on a same processing node 1 over local buses (e.g.,PCIe buses) using a protocol like Transmission Control Protocol (TCP).

FIG. 13 is a message sequence chart 1300 illustrating one implementationof executing a first set of functions in configuration files and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) on one or more of the reconfigurable processors (RP)142 a and executing a second set of functions and/or data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) in the configuration files on the second host processor 102 n.At operation one, the compiler 112 a receives an application 1302 forcompilation. At operation two, the compiler 112 a compiles theapplication 1302 to generate one or more configuration files 1312. Theconfiguration files 1312 include a plurality of functions. The pluralityof functions includes a first set of functions 1314 and a second set offunctions 1324. Examples of functions in the plurality of functionsinclude non-linearities like Rectified Linear Unit (ReLU) and itsvariants (e.g., leaky ReLU), hyperbolic tangent, sigmoid, and softmax,element-wise addition, matrix multiplication (e.g., General MatrixMultiply (GeMM)), layer normalization (e.g., batch normalization), lossfunctions like cross-entropy, and tensor shape modifiers like transpose.At operation three, the compiler 112 a sends the configuration files1312 to the runtime logic 122 a for execution. At operation four, theruntime logic 122 a loads the first set of functions 1314 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) and the second set of functions 1324 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) on one or more of the reconfigurable processors142 a. At operation five, the reconfigurable processors 142 a processthe first set of functions 1314 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) and generatea first set of outputs (e.g., vectors, tensors). The reconfigurableprocessors 142 a transmit functions in the second set of functions 1324and/or data therefor (e.g., weights, coefficients, vectors, tensors(image data, audio data, natural language processing (NLP data), controldata (e.g., control tokens)) to the second host processor 102 n usingone or more reconfigurable processors-to-host processor buffers. This isreferred to herein as “reconfigurable processor-to-host processorworkload sharing.” In one implementation, data on which the functions inthe second set of functions 1324 are executed is transmitted to thesecond host processor 102 n using the reconfigurable processors-to-hostprocessor buffers. In some implementations, respective ones of thereconfigurable processors-to-host processor buffers are used to transmitrespective functions in the second set of functions 1324 and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) to the second host processor 102 n. One exampleworkload sharing flow includes using one or more of the reconfigurableprocessor sender buffers 412 a and one or more of the host receiverbuffers 202 n. At operation six, the reconfigurable processors 142 atransmit the functions in the second set of functions 1324 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) to the reconfigurable processor sender buffers412 a. At operation seven, the reconfigurable processor sender buffers412 a transmit the functions in the second set of functions 1324 and/orthe data therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) to the host receiver buffers 202 n. At operationeight, the host receiver buffers 202 n transmit the functions in thesecond set of functions 1324 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) to thesecond host processor 102 n. At operation nine, the second hostprocessor 102 n executes the functions in the second set of functions1324 and/or the data therefor (e.g., weights, coefficients, vectors,tensors (image data, audio data, natural language processing (NLP data),control data (e.g., control tokens)) to generate a second set of outputs(or results 1334) (e.g., vectors, tensors). The second host processor102 n transmits the results 1334 to one or more of the reconfigurableprocessors 142 a using one or more host processor-to-reconfigurableprocessors buffers. In some implementations, respective ones of the hostprocessor-to-reconfigurable processors buffers are used to transmitrespective results of executing respective functions in the second setof functions 1324 and/or data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) to the reconfigurableprocessors 142 a. One workload sharing flow includes using one or moreof the host sender buffers 212 n and one or more of the reconfigurableprocessor receiver buffers 402 a. At operation ten, the second hostprocessor 102 n transmits the results 1334 to the host sender buffers212 n. At operation eleven, the host sender buffers 212 n transmit theresults 1334 to the reconfigurable processor receiver buffers 402 a. Atoperation twelve, the reconfigurable processor receiver buffers 402 atransmit the results 1334 to the reconfigurable processors 142 a. Insome implementations, one or more functions in the first set offunctions 1314 waits for results of execution of one or more functionsin the second set of functions 1324 on the second host processor 102 nto combine the results with results of execution of one or morefunctions in the first set of functions 1314 and/or the data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) on the reconfigurable processors 142 a. In otherimplementations, the first set of functions 1314 and the second set offunctions 1324 operate separately and in parallel. In oneimplementation, one or more functions in the second set of functions1324 daisy chains the results to one or more functions in the first setof functions 1314, and vice-versa. In another implementation, one ormore functions in the second set of functions 1324 executes for acertain number of iterations before returning the results to thereconfigurable processors 142 a. Other implementations may perform theoperations in different orders and/or with different, fewer, oradditional operations than the ones illustrated in FIG. 13. Multipleoperations can be combined in some implementations.

One skilled in the art will appreciate that, in FIG. 13, operations six,seven, eight, ten, eleven, and twelve comprise streaming network packetsbetween one or more reconfigurable processors (e.g., RPs 142 a) on thefirst processing node and a host processor (e.g., host 102 n) on thesecond processing node over the network fabric 136 (e.g., Ethernet,InfiniBand (IB)) using protocols like RDMA over Converged Ethernet(RoCE), TCP, User Datagram Protocol (UDP), and Quick UDP InternetConnections (QUIC).

Having described the reconfigurable processor-to-host processor workloadsharing, the discussion now turns to the reconfigurableprocessor-to-reconfigurable processor workload sharing disclosed by thetechnology disclosed.

Reconfigurable Processor-to-Reconfigurable Processor Workload Sharing

FIG. 14A shows sender and receiver buffers used by individualreconfigurable processors in the reconfigurable processors 142 a.Reconfigurable processor 1 (RP 1) receiver buffers 1402 a andreconfigurable processor 1 (RP 1) sender buffers 1412 a are used by afirst reconfigurable processor in the reconfigurable processors 142 a toreceive data from and send data to another host processor orreconfigurable processor in the data center 100. Reconfigurableprocessor n (RP n) receiver buffers 1422 a and reconfigurable processorn (RP n) sender buffers 1432 a are used by a second reconfigurableprocessor in the reconfigurable processors 142 a to receive data fromand send data to another host processor or reconfigurable processor inthe data center 100. The reconfigurable processor 1 receiver buffers1402 a, the reconfigurable processor 1 sender buffers 1412 a, thereconfigurable processor n receiver buffers 1422 a, and thereconfigurable processor n sender buffers 1432 a are located in thereconfigurable processor memory 162 a.

FIG. 14B shows sender and receiver buffers used by individualreconfigurable processors in the reconfigurable processors 142 n.Reconfigurable processor 1 (RP 1) receiver buffers 1402 n andreconfigurable processor 1 (RP 1) sender buffers 1412 n are used by afirst reconfigurable processor in the reconfigurable processors 142 n toreceive data from and send data to another host processor orreconfigurable processor in the data center 100. Reconfigurableprocessor n (RP n) receiver buffers 1422 n and reconfigurable processorn (RP n) sender buffers 1432 n are used by a second reconfigurableprocessor in the reconfigurable processors 142 n to receive data fromand send data to another host processor or reconfigurable processor inthe data center 100. The reconfigurable processor 1 receiver buffers1402 n, the reconfigurable processor 1 sender buffers 1412 n, thereconfigurable processor n receiver buffers 1422 n, and thereconfigurable processor n sender buffers 1432 n are located in thereconfigurable processor memory 162 n.

Intra-Node Processing

FIG. 15 is a message sequence chart 1500 illustrating one implementationof executing a first set of functions in configuration files and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) on a first reconfigurable processor in thereconfigurable processors 142 a and executing a second set of functionsin the configuration files and/or data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on a secondreconfigurable processor in the reconfigurable processors 142 a. In FIG.15, the first reconfigurable processor is identified as “RP 1” and thesecond reconfigurable processor is identified as “RP N.” Note that thefirst reconfigurable processor and the second reconfigurable processorare operatively coupled to a same processing node, i.e., the firstprocessing node. This is referred to herein as “intra-node processing.”At operation one, the compiler 112 a receives an application 1502 forcompilation. At operation two, the compiler 112 a compiles theapplication 1502 to generate one or more configuration files 1512. Theconfiguration files 1512 include a plurality of functions. The pluralityof functions includes a first set of functions 1514 and a second set offunctions 1524. Examples of functions in the plurality of functionsinclude non-linearities like Rectified Linear Unit (ReLU) and itsvariants (e.g., leaky ReLU), hyperbolic tangent, sigmoid, and softmax,element-wise addition, matrix multiplication (e.g., General MatrixMultiply (GeMM)), layer normalization (e.g., batch normalization), lossfunctions like cross-entropy, and tensor shape modifiers like transpose.At operation three, the compiler 112 a sends the configuration files1512 to the runtime logic 122 a for execution. At operation four, theruntime logic 122 a loads the first set of functions 1514 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) and the second set of functions 1524 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) on the first reconfigurable processor. Atoperation five, the first reconfigurable processor processes the firstset of functions 1514 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) andgenerates a first set of outputs (e.g., vectors, tensors). The firstreconfigurable processor transmits functions in the second set offunctions 1524 and/or data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) to the secondreconfigurable processor using one or more reconfigurableprocessors-to-reconfigurable processors buffers. This is referred toherein as “reconfigurable processor-to-reconfigurable processor workloadsharing.” In one implementation, data on which the functions in thesecond set of functions 1524 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) are executedis transmitted to the second reconfigurable processor using thereconfigurable processors-to-reconfigurable processors buffers. In someimplementations, respective ones of the reconfigurableprocessors-to-reconfigurable processors buffers are used to transmitrespective functions in the second set of functions 1524 and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) to the second reconfigurable processor. One exampleworkload sharing flow includes using one or more of the reconfigurableprocessor 1 (RP 1) sender buffers 1412 a and one or more of thereconfigurable processor N (RP N) receiver buffers 1422 a. At operationsix, the first reconfigurable processor transmits the functions in thesecond set of functions 1524 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) to thereconfigurable processor 1 sender buffers 1412 a. At operation seven,the reconfigurable processor 1 sender buffers 1412 a transmit thefunctions in the second set of functions 1524 and/or the data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) to the reconfigurable processor N receiver buffers 1422 a. Atoperation eight, the reconfigurable processor N receiver buffers 1422 atransmit the functions in the second set of functions 1524 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) to the second reconfigurable processor. Atoperation nine, the second reconfigurable processor executes thefunctions in the second set of functions 1524 and/or the data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) to generate a second set of outputs (or results 1534) (e.g.,vectors, tensors). The second reconfigurable processor transmits theresults 1534 to the first reconfigurable processor using one or more ofthe reconfigurable processors-to-reconfigurable processors buffers. Insome implementations, respective ones of the reconfigurableprocessors-to-reconfigurable processors buffers are used to transmitrespective results of executing respective functions in the second setof functions 1524 and/or data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) to the firstreconfigurable processor. One workload sharing flow includes using oneor more of the reconfigurable processor N (RP N) sender buffers 1432 aand one or more of the reconfigurable processor 1 (RP 1) receiverbuffers 1402 a. At operation ten, the second reconfigurable processortransmits the results 1534 to the reconfigurable processor N senderbuffers 1432 a. At operation eleven, the reconfigurable processor Nsender buffers 1432 a transmit the results 1534 to the reconfigurableprocessor 1 receiver buffers 1402 a. At operation twelve, thereconfigurable processor 1 receiver buffers 1402 a transmit the results1534 to the first reconfigurable processor. In some implementations, oneor more functions in the first set of functions 1514 waits for resultsof execution of one or more functions in the second set of functions1524 on the second reconfigurable processor to combine the results withresults of execution of one or more functions in the first set offunctions 1514 and/or the data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) on the firstreconfigurable processor. In other implementations, the first set offunctions 1514 and the second set of functions 1524 operate separatelyand in parallel. In one implementation, one or more functions in thesecond set of functions 1524 daisy chains the results to one or morefunctions in the first set of functions 1514, and vice-versa. In anotherimplementation, one or more functions in the second set of functions1524 executes for a certain number of iterations before returning theresults to the first reconfigurable processor. Other implementations mayperform the operations in different orders and/or with different, fewer,or additional operations than the ones illustrated in FIG. 15. Multipleoperations can be combined in some implementations.

One skilled in the art will appreciate that, in FIG. 15, operations six,seven, eight, ten, eleven, and twelve comprise streaming network packetsbetween reconfigurable processors on a same processing node 1 over localbuses (e.g., PCIe buses) using a protocol like Transmission ControlProtocol (TCP).

Inter-Node Processing

FIG. 16 is a message sequence chart 1600 illustrating one implementationof executing a first set of functions in configuration files and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) on a first reconfigurable processor in thereconfigurable processors 142 a and executing a second set of functionsin the configuration files and/or data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on a firstreconfigurable processor in the reconfigurable processors 142 n. In FIG.16, the first reconfigurable processor in the reconfigurable processors142 a is identified as “RP 1” and the first reconfigurable processor inthe reconfigurable processors 142 n is identified as “RP 1′.” Note thatthe first reconfigurable processor in the reconfigurable processors 142a and the first reconfigurable processor in the reconfigurableprocessors 142 n are operatively coupled to different processing nodes,i.e., the first processing node and the second processing node. This isreferred to herein as “inter-node processing.” At operation one, thecompiler 112 a receives an application 1602 for compilation. Atoperation two, the compiler 112 a compiles the application 1602 togenerate one or more configuration files 1612. The configuration files1612 include a plurality of functions. The plurality of functionsincludes a first set of functions 1614 and a second set of functions1624. Examples of functions in the plurality of functions includenon-linearities like Rectified Linear Unit (ReLU) and its variants(e.g., leaky ReLU), hyperbolic tangent, sigmoid, and softmax,element-wise addition, matrix multiplication (e.g., General MatrixMultiply (GeMM)), layer normalization (e.g., batch normalization), lossfunctions like cross-entropy, and tensor shape modifiers like transpose.At operation three, the compiler 112 a sends the configuration files1612 to the runtime logic 122 a for execution. At operation four, theruntime logic 122 a loads the first set of functions 1614 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) and the second set of functions 1624 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) on the first reconfigurable processor in thereconfigurable processors 142 a. At operation five, the firstreconfigurable processor in the reconfigurable processors 142 aprocesses the first set of functions 1614 and/or the data therefor(e.g., weights, coefficients, vectors, tensors (image data, audio data,natural language processing (NLP data), control data (e.g., controltokens)) and generates a first set of outputs (e.g., vectors, tensors).The first reconfigurable processor in the reconfigurable processors 142a transmits functions in the second set of functions 1624 and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) to the first reconfigurable processor in thereconfigurable processors 142 n using one or more reconfigurableprocessors-to-reconfigurable processors buffers. This is referred toherein as “reconfigurable processor-to-reconfigurable processor workloadsharing.” In one implementation, data on which the functions in thesecond set of functions 1624 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) are executedis transmitted to the first reconfigurable processor in thereconfigurable processors 142 n using the reconfigurableprocessors-to-reconfigurable processors buffers. In someimplementations, respective ones of the reconfigurableprocessors-to-reconfigurable processors buffers are used to transmitrespective functions in the second set of functions 1624 and/or datatherefor (e.g., weights, coefficients, vectors, tensors (image data,audio data, natural language processing (NLP data), control data (e.g.,control tokens)) to the first reconfigurable processor in thereconfigurable processors 142 n. One example workload sharing flowincludes using one or more of the reconfigurable processor 1 (RP 1)sender buffers 1412 a and one or more of the reconfigurable processor 1′(RP 1′) receiver buffers 1402 n. At operation six, the firstreconfigurable processor in the reconfigurable processors 142 atransmits the functions in the second set of functions 1624 and/or thedata therefor (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) to the reconfigurable processor 1 sender buffers1412 a. At operation seven, the reconfigurable processor 1 senderbuffers 1412 a transmit the functions in the second set of functions1624 and/or the data therefor (e.g., weights, coefficients, vectors,tensors (image data, audio data, natural language processing (NLP data),control data (e.g., control tokens)) to the reconfigurable processor 1′receiver buffers 1402 n. At operation eight, the reconfigurableprocessor 1′ receiver buffers 1402 n transmit the functions in thesecond set of functions 1624 and/or the data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) to the firstreconfigurable processor in the reconfigurable processors 142 n. Atoperation nine, the first reconfigurable processor in the reconfigurableprocessors 142 n executes the functions in the second set of functions1624 and/or the data therefor (e.g., weights, coefficients, vectors,tensors (image data, audio data, natural language processing (NLP data),control data (e.g., control tokens)) to generate a second set of outputs(or results 1634) (e.g., vectors, tensors). The first reconfigurableprocessor in the reconfigurable processors 142 n transmits the results1634 to the first reconfigurable processor in the reconfigurableprocessors 142 a using one or more of the reconfigurableprocessors-to-reconfigurable processors buffers. In someimplementations, respective ones of the reconfigurableprocessors-to-reconfigurable processors buffers are used to transmitrespective results of executing respective functions in the second setof functions 1624 and/or data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) to the firstreconfigurable processor in the reconfigurable processors 142 a. Oneworkload sharing flow includes using one or more of the reconfigurableprocessor 1′ (RP 1′) sender buffers 1412 n and one or more of thereconfigurable processor 1 (RP 1) receiver buffers 1402 a. At operationten, the first reconfigurable processor in the reconfigurable processors142 n transmits the results 1634 to the reconfigurable processor 1′sender buffers 1412 n. At operation eleven, the reconfigurable processor1′ sender buffers 1412 n transmit the results 1634 to the reconfigurableprocessor 1 receiver buffers 1402 a. At operation twelve, thereconfigurable processor 1 receiver buffers 1402 a transmit the results1634 to the first reconfigurable processor in the reconfigurableprocessors 142 a. In some implementations, one or more functions in thefirst set of functions 1614 waits for results of execution of one ormore functions in the second set of functions 1624 on the firstreconfigurable processor in the reconfigurable processors 142 n tocombine the results with results of execution of one or more functionsin the first set of functions 1614 and/or the data therefor (e.g.,weights, coefficients, vectors, tensors (image data, audio data, naturallanguage processing (NLP data), control data (e.g., control tokens)) onthe first reconfigurable processor in the reconfigurable processors 142a. In other implementations, the first set of functions 1614 and thesecond set of functions 1624 operate separately and in parallel. In oneimplementation, one or more functions in the second set of functions1624 daisy chains the results to one or more functions in the first setof functions 1614, and vice-versa. In another implementation, one ormore functions in the second set of functions 1624 executes for acertain number of iterations before returning the results to the firstreconfigurable processor in the reconfigurable processors 142 a. Otherimplementations may perform the operations in different orders and/orwith different, fewer, or additional operations than the onesillustrated in FIG. 16. Multiple operations can be combined in someimplementations.

One skilled in the art will appreciate that, in FIG. 16, operations six,seven, eight, ten, eleven, and twelve comprise streaming network packetsbetween reconfigurable processors on different processing nodes 1 and nover the network fabric 136 (e.g., Ethernet, InfiniBand (IB)) usingprotocols like RDMA over Converged Ethernet (RoCE), TCP, User DatagramProtocol (UDP), and Quick UDP Internet Connections (QUIC).

Having described the reconfigurable processor-to-reconfigurableprocessor workload sharing, the discussion now turns to the asynchronoustensor streaming disclosed by the technology disclosed.

Asynchronous Tensor Streaming

FIG. 17A is a message sequence chart 1700A illustrating oneimplementation of asynchronous tensor streaming in which a next tensoris buffered while a reconfigurable processor is processing a currenttensor. A reconfigurable processor in the data center 100 (e.g., one ormore of the reconfigurable processors 142 a) is configured to executeone or more configuration files using a series of data units 1712. Inone implementation, the series of data units 1712 includes a sequence oftensors 1 to N. A first plurality of buffers 1704 is configured toreceive data units in the series of data units 1712 from a source memory1702 (e.g., host memory 134 a, host memory 134 n), and to stream thedata units to the reconfigurable processor for processing. Examples ofbuffers in the first plurality of buffers 1704 First-In, First-Out(FIFO) buffers, First-In, Last-Out (FILO) buffers, Last-In, First-Out(LIFO) buffers, Last-In, Last-Out (LILO) buffers, and circular buffers.The buffers in the first plurality of buffers 1704 can be of size 8bytes, 16 bytes, 32 bytes, 64 bytes, 128 bytes, 256 bytes, and so on, orany convenient size appropriate for the transfer of data between thehost processor, the network interface controllers, and thereconfigurable processors. A second plurality of buffers 1706 isconfigured to stream results of processing the data units from thereconfigurable processor, and to send the results to a destinationmemory 1708 (e.g., reconfigurable processor memory 162 a, reconfigurableprocessor memory 162 n) for storage. Examples of buffers in the secondplurality of buffers 1706 include First-In, First-Out (FIFO) buffers,First-In, Last-Out (FILO) buffers, Last-In, First-Out (LIFO) buffers,Last-In, Last-Out (LILO) buffers, and circular buffers. The buffers inthe second plurality of buffers 1706 can be of size 8 bytes, 16 bytes,32 bytes, 64 bytes, 128 bytes, 256 bytes, and so on, or any convenientsize appropriate for the transfer of data between the host processor,the network interface controllers, and the reconfigurable processors.

A runtime logic (e.g., runtime 122 a, runtime 122 n) is configured tocause the buffers in the first plurality of buffers 1704 to receive anext data unit in the series of data units 1712 from the source memory1702 while the reconfigurable processor processes a current data unit inthe series of data units 1712. The runtime logic is further configuredto stream the next data unit to the reconfigurable processor forprocessing after the buffers in the second plurality of buffers 1706stream results of processing the current data unit from thereconfigurable processor.

Turning to the example illustrated in FIG. 17A. Consider that tensor 1is the current data unit and tensors 2 and 3 are next data units. Attimestep one, the buffers in the first plurality of buffers 1704 receivetensor 1 from the source memory 1702. At timestep two, the buffers inthe first plurality of buffers 1704 stream tensor 1 to thereconfigurable processor. At timestep three, the reconfigurableprocessor starts processing tensor 1. While the reconfigurable processoris processing tensor 1, the buffers in the first plurality of buffers1704 receive tensors 2 and 3 from the source memory 1702 at timestepsfour and five, respectively. At timestep six, the reconfigurableprocessor streams results of processing tensor 1 (result 1) to thebuffers in the second plurality of buffers 1706. At timestep seven, thebuffers in the second plurality of buffers 1706 stream the results ofprocessing tensor 1 to the destination memory 1708 for storage. Attimestep eight, the buffers in the first plurality of buffers 1704stream tensor 2 to the reconfigurable processor. In otherimplementations, streaming of tensor 2 from the buffers in the firstplurality of buffers 1704 to the reconfigurable processor precedes thestreaming of the results of processing tensor 1 from the buffers in thesecond plurality of buffers 1706 to the destination memory 1708. Otherimplementations may perform the steps in different orders and/or withdifferent, fewer, or additional steps than the ones illustrated in FIG.17A. In some implementations, processing of tensors in one or moreprevious timesteps/iterations (e.g., tensors 2 and 3) by thereconfigurable processors 142 a overlaps with the processing of a tensorin a current timestep/iteration (e.g., tensor 1) by the reconfigurableprocessors 142 a. This is referred to herein as “meta-pipelining.”Multiple steps can be combined in some implementations.

FIG. 17B is a message sequence chart 1700B illustrating oneimplementation of asynchronous tensor streaming in which a next tensoris buffered before a reconfigurable processor processes a currenttensor. The runtime logic is further configured to cause the buffers inthe first plurality of buffers 1704 to receive the next data unit fromthe source memory 1702 before the reconfigurable processor startsprocessing the current data unit.

Turning to the example illustrated in FIG. 17B. At timestep one, thebuffers in the first plurality of buffers 1704 receive tensor 1 from thesource memory 1702. At timestep two, the buffers in the first pluralityof buffers 1704 stream tensor 1 to the reconfigurable processor. Beforethe reconfigurable processor starts processing tensor 1, the buffers inthe first plurality of buffers 1704 receive tensors 2 and 3 from thesource memory 1702 at timesteps three and four, respectively. Attimestep five, the reconfigurable processor starts processing tensor 1.At timestep six, the reconfigurable processor streams results ofprocessing tensor 1 (result 1) to the buffers in the second plurality ofbuffers 1706. At timestep seven, the buffers in the second plurality ofbuffers 1706 stream the results of processing tensor 1 to thedestination memory 1708 for storage. At timestep eight, the buffers inthe first plurality of buffers 1704 stream tensor 2 to thereconfigurable processor. In other implementations, streaming of tensor2 from the buffers in the first plurality of buffers 1704 to thereconfigurable processor precedes the streaming of the results ofprocessing tensor 1 from the buffers in the second plurality of buffers1706 to the destination memory 1708. Other implementations may performthe steps in different orders and/or with different, fewer, oradditional steps than the ones illustrated in FIG. 17B. Multiple stepscan be combined in some implementations.

FIG. 17C is a message sequence chart 1700C illustrating oneimplementation of asynchronous tensor streaming in which a next tensoris buffered after a reconfigurable processor has processed a currenttensor. The runtime logic is further configured to cause the buffers inthe first plurality of buffers 1704 to receive the next data unit fromthe source memory 1702 after the buffers in the second plurality ofbuffers 1706 stream the results of processing the current data unit fromthe reconfigurable processor.

Turning to the example illustrated in FIG. 17C. At timestep one, thebuffers in the first plurality of buffers 1704 receive tensor 1 from thesource memory 1702. At timestep two, the buffers in the first pluralityof buffers 1704 stream tensor 1 to the reconfigurable processor. Attimestep three, the reconfigurable processor starts processing tensor 1.At timestep four, the reconfigurable processor streams results ofprocessing tensor 1 (result 1) to the buffers in the second plurality ofbuffers 1706. After the buffers in the second plurality of buffers 1706stream the results of processing tensor 1 from the reconfigurableprocessor, the buffers in the first plurality of buffers 1704 receivetensors 2 and 3 from the source memory 1702 at timesteps five and six,respectively. At timestep seven, the buffers in the second plurality ofbuffers 1706 stream the results of processing tensor 1 to thedestination memory 1708 for storage. At timestep eight, the buffers inthe first plurality of buffers 1704 stream tensor 2 to thereconfigurable processor. In other implementations, streaming of tensor2 from the buffers in the first plurality of buffers 1704 to thereconfigurable processor precedes the streaming of the results ofprocessing tensor 1 from the buffers in the second plurality of buffers1706 to the destination memory 1708. Other implementations may performthe steps in different orders and/or with different, fewer, oradditional steps than the ones illustrated in FIG. 17C. Multiple stepscan be combined in some implementations.

Having described the asynchronous tensor streaming, the discussion nowturns to how the technology disclosed executes configuration files onreconfigurable processors that are on different processing nodes in thedata center 100. This is referred to herein as “inter-node execution ofconfiguration files.”

Inter-Node Execution of Configuration Files

FIG. 18 is a message sequence chart 1800 illustrating one implementationof executing configuration files on reconfigurable processors that areon different processing nodes in the data center 100. This is referredto herein as “inter-node execution of configuration files.” The datacenter 100 comprises a pool of reconfigurable dataflow resources. Thepool of reconfigurable dataflow resources includes a plurality ofprocessing nodes (e.g., processing nodes 1 to n). Respective processingnodes in the plurality of processing nodes are operatively coupled torespective pluralities of reconfigurable processors (RPs) and respectivepluralities of buffers. The respective processing nodes are alsooperatively coupled to respective host processors. The respectiveprocessing nodes are also operatively coupled to respective pluralitiesof Network Interface Controllers (NICs) or Smart Network InterfaceControllers (SmartNICs).

In one implementation, buffers in the respective pluralities of buffersare located in respective memories of the respective pluralities ofreconfigurable processors. Examples of the respective memories of therespective pluralities of reconfigurable processors include off-chipand/or on-chip memories like DRAM, NAND flash, SRAM, latches, flops,bypass networks, and registers. In another implementation, the buffersare located in respective memories of NICs or SmartNICs in therespective pluralities of NICs or SmartNICs. In yet anotherimplementation, the buffers are located in respective memories of hostprocessors (e.g., RAM/ROM, caches) in the respective host processors. Inother implementations, the buffers can be located in or attached to anynetwork component of the data center 100 such as PCIe buses, Double DataRate (DDR) channels, Dual In-Line Memory Modules (DIMMs), routers, andswitches. The buffers can be First-In, First-Out (FIFO) buffers,First-In, Last-Out (FILO) buffers, Last-In, First-Out (LIFO) buffers,Last-In, Last-Out (LILO) buffers, or circular buffers. The buffers canbe of size 8 bytes, 16 bytes, 32 bytes, 64 bytes, 128 bytes, 256 bytes,and so on, or any convenient size appropriate for the transfer of databetween the host processor, the network interface controllers, and thereconfigurable processors.

A compiler 1812 compiles applications 1802 (operation one) and generatesconfiguration files 1822 (operation two). The configuration files 1822specify configurations of virtual dataflow resources 1824 required toexecute the configuration files 1822. In one implementation, the virtualdataflow resources 1824 include a first virtual reconfigurable processor1824 a 1 in a first virtual processing node 1824 a, a second virtualreconfigurable processor 1824 b 1 in a second virtual processing node1824 b, and virtual buffers 1824 c that stream data between the firstvirtual reconfigurable processor 1824 a 1 and the second virtualreconfigurable processor 1824 b 1. The virtual buffers 1824 c comprisefirst virtual SmartNIC buffers 1824 c 1 and second virtual SmartNICbuffers 1824 c 2.

A runtime processor 1832 is operatively coupled to the pool ofreconfigurable dataflow resources and configured to receive theconfiguration files 1822 (operation three). The runtime processor 1832comprises a runtime logic 1842 and an allocation logic 1844. Theallocation logic 1844 is configured to allocate reconfigurable dataflowresources in the pool of reconfigurable dataflow resources to thevirtual dataflow resources 1824 (operation four). The allocatedreconfigurable dataflow resources include a first processing node in therespective processing nodes allocated to the first virtual processingnode 1824 a, a second processing node in the respective processing nodesallocated to the second virtual processing node 1824 b, a firstreconfigurable processor, operatively coupled to the first processingnode, allocated to the first virtual reconfigurable processor 1824 a 1,a second reconfigurable processor operatively coupled to the secondprocessing node allocated to the second virtual reconfigurable processor1824 b 1, and a first plurality of buffers, operatively coupled to thefirst processing node, and a second plurality of buffers, operativelycoupled to the second processing node, allocated to the virtual buffers1824 c. The runtime logic 1842 is configured to execute theconfiguration files 1822 using the allocated reconfigurable dataflowresources (operation five).

The discussion now turns to how buffers can be allocated for inter-nodestreaming of configuration data (e.g., bit stream) by mapping physicalmemory addresses of the buffers to memories of different networkcomponents in the data center 100 (e.g., host memories, reconfigurableprocessor memories, NIC memories, SmartNIC memories, PCIe bus memories,DDR channel memories, DIMM memories, etc.).

Buffer Allocation

The buffers are programmable and can be allocated by specifying physicalmemory addresses. The physical memory addresses of the buffers specifymemory locations of the buffers. The physical memory addresses of thebuffers can be designated by the host processors and/or by thereconfigurable processors. The configurations of the virtual buffers1824 c specify virtual memory segments of the buffers allocated forexecution of the applications 1802 (e.g., the first and second pluralityof buffers), including virtual address spaces (e.g., starting or baseaddresses) of the virtual memory segments and sizes of the virtualaddress spaces (e.g., sizes of the memory blocks in bytes). The runtimeprocessor 1832 maps the virtual address spaces of the virtual memorysegments to physical address spaces of physical memory segments inmemory where the allocated buffers are located. The memory can be hostprocessor memory, reconfigurable processor memory (off-chip or on-chip),NIC memory, SmartNIC memory, PCIe memory, DMA memory, DIMM memory, orany other network component memory in the data center 100.

FIG. 19 shows one implementation of memory mapping 1900 the virtualbuffers 1824 c to allocated buffers 1902/physical buffers 1902 locatedin respective physical memories of example reconfigurable dataflowresources such as SmartNIC one (SmartNIC 1) memory, SmartNIC two(SmartNIC 2) memory, reconfigurable processor one (RP 1) memory,reconfigurable processor two (RP 2) memory, PCIe 1 memory, DMA 1 memory,and host processor 1 memory. FIG. 19 shows that Control and StatusRegisters (CSRs) 1980 of the example reconfigurable dataflow resourcesare used for memory mapping the virtual buffers 1824 c in a virtualmemory space to physical memory space. CSRs 1913, 1923, 1933, 1943,1953, 1963, and 1973 in the allocated physical element (e.g., SmartNIC,RP, DMA engine of PCIe device, etc.) are used to map the applicationvirtual buffer addresses to the appropriate physical addresses by havingthe runtime logic program them. (e.g., SmartNIC 1 buffers 1912, SmartNIC2 buffers 1922, RP 1 buffers 1932, RP 2 buffers 1942, PCIe 1 buffers1952, DMA 1 buffers 1962, host 1 buffers 1972) to the allocated buffers1902 in a contiguous physical memory space (e.g., SmartNIC 1 buffers1914 (first range of physical memory addresses), SmartNIC 2 buffers 1924(second range of physical memory addresses), RP 1 buffers 1934 (thirdrange of physical memory addresses), RP 2 buffers 1944 (fourth range ofphysical memory addresses), PCIe 1 buffers 1954 (fifth range of physicalmemory addresses), DMA 1 buffers 1964 (sixth range of physical memoryaddresses), host 1 buffers 1974 (seventh range of physical memoryaddresses)).

The discussion now turns to how buffer allocation is done independentlyfor multiple applications being executed simultaneously or in parallelon a given set of reconfigurable dataflow resources (e.g.,reconfigurable processors, NICs, SmartNICs, PCIe buses, DMA channels),and how allocated buffers are kept isolated on anapplication-by-application basis.

Application-Wise Buffer Allocation and Isolation

In one implementation, the runtime processor 1832 configures control andstatus registers of the reconfigurable dataflow resources withconfiguration data (e.g., bit stream) identifying the mapping betweenthe virtual address spaces and the physical address spaces for theconfiguration files 1822 to access the physical memory segments duringexecution of the applications 1802. In some implementations, a first setof the physical memory segments mapped to buffers allocated to a firstone of the applications 1802 are different from a second set of thephysical memory segments mapped to buffers allocated to a second one ofthe applications 1802. Also, access of the buffers allocated to thefirst one of the applications 1802 is confined to the first set of thephysical memory segments, and access of the buffers allocated to thesecond one of the applications 1802 is confined to the second set of thephysical memory segments.

In some implementations, the reconfigurable processors have respectivepluralities of buffers for respective applications such that a firstplurality of buffers can be used to stream configuration data (e.g., bitstream) to execute configuration files for a first application, a secondplurality of buffers can be used to stream configuration data (e.g., bitstream) to execute configuration files for a second application, a thirdplurality of buffers can be used to stream configuration data (e.g., bitstream) to execute configuration files for a third application, and soon. The configuration files for the first, second, and thirdapplications can be executed in parallel or sequence using the first,second, and third plurality of buffers, respectively. In oneimplementation, the configuration files for the first, second, and thirdapplications can be executed, in parallel or in sequence, on a singlereconfigurable processor using the first, second, and third plurality ofbuffers, respectively. In another implementation, the configurationfiles for the first, second, and third applications can be executed, inparallel or in sequence, across reconfigurable processors on a sameprocessing node using the first, second, and third plurality of buffers,respectively, such that, in some implementations, each of the first,second, and third plurality of buffers includes a set of sender (TX)buffers and receiver (RX) buffers for each reconfigurable processor orNIC or SmartNIC on the same processing node used to execute theconfiguration files. In yet another implementation, the configurationfiles for the first, second, and third applications can be executed, inparallel or in sequence, across reconfigurable processors on differentprocessing nodes using the first, second, and third plurality ofbuffers, respectively, such that, in some implementations, each of thefirst, second, and third plurality of buffers includes a set of sender(TX) buffers and receiver (RX) buffers for each reconfigurable processoror NIC or SmartNIC on the different processing nodes used to execute theconfiguration files.

In one implementation, the runtime processor 1832 runs on each hostprocessor in the data center 100 and provides unified access to the poolof reconfigurable dataflow resources in the data center 100. Additionaldetails about how the allocation logic 1844 spans the userspace andkernel space of a host processor on which a runtime processor or runtimelogic runs can be found in U.S. Nonprovisional patent application Ser.No. 16/922,975, filed Jul. 7, 2020, entitled, “RUNTIME VIRTUALIZATION OFRECONFIGURABLE DATAFLOW RESOURCES,” (Attorney Docket No. SBNV 1026-1),which is incorporated herein by reference (specific reference is made tothe runtime library 312, the kernel module 322, the resource manager471, the device driver 474, and other allocation logic and components inthe application incorporated by reference).

The discussion now turns to how various aspects of the technologydisclosed described in this application can be executed without the useof hosts or host processors. Such implementations are referred to hereinas “host-less implementations.”

Hypervisor and Host-Less Implementations

In another implementation, the runtime processor 1832 runs on eachreconfigurable processor in the data center 100 and provides unifiedaccess to the pool of reconfigurable dataflow resources in the datacenter 100. In yet another implementation, the runtime processor 1832runs as a hypervisor only on a subset of the host processors in the datacenter 100 (e.g., only on one host processor). In yet anotherimplementation, the runtime processor 1832 runs as a hypervisor only ona subset of the reconfigurable processors in the data center 100 (e.g.,only on one reconfigurable processor).

FIG. 20 shows an architectural level schematic 2000 of oneimplementation of the data center 100 in which the processing nodes ofthe data center 100 do not include host processors. The implementationshown in the architectural level schematic 2000 is configured to executeother implementations discussed in this application (e.g., intra-nodeprocessing, inter-node execution of configuration files), except thatthe other implementations are executed without using the hostprocessors. In many host-less implementations, functionalities that areotherwise performed by host processors are instead performed by thereconfigurable processors in the data center 100. Some examples offunctionalities performed by the reconfigurable processors in host-lessimplementations include hosting the compiler 1812, compiling theapplications 1802, generating the configuration files 1822, generatingconfigurations of the virtual dataflow resources 1824, hosting theruntime processor 1832, memory mapping, resource allocation (e.g.,designating and allocating physical memory addresses of the buffers andother reconfigurable dataflow resources), execution of the configurationfiles 1822, parsing incoming network packets and running anomalydetection in ultra-low and deterministic latency, etc.). In otherhost-less implementations, the functionalities that are otherwiseperformed by the host processors are obviated by other networkcomponents in the data center 100, for example, by the SmartNICs thatcomprise microcontrollers to locally trigger host-like commands withoutrequiring an external host.

In the hypervisor and the host-less implementations, the runtimeprocessor 1832 can be considered a distributed runtime processor, adistributed runtime logic, a distributed resource manager, and/or adistributed resource allocator that provides unified access to the poolof reconfigurable dataflow resources in the data center 100.

The discussion now turns to how, for efficient execution of theconfiguration files, the technology disclosed uses buffers to stream,over a network fabric, configuration data (e.g., bit stream) betweenreconfigurable processors that are on different processing nodes in thedata center 100. This is referred to herein as “buffer-based inter-nodestreaming of configuration data (e.g., bit stream) over network fabric.”

Buffer-Based Inter-Node Streaming of Configuration Data (e.g., BitStream) Over Network Fabric

FIG. 21 is a message sequence chart 2100 illustrating one implementationof buffer-based inter-node streaming of configuration data (e.g., bitstream) over the network fabric 136. In the implementation illustratedin FIG. 21, buffers used for the inter-node streaming, i.e., senderbuffers 2176 a, receiver buffers 2178 a, sender buffers 2176 n, andreceiver buffers 2178 n, are located in respective memories of theSmartNIC devices 132 a and 132 n. However, these buffers can be locatedin any network component of the data center 100 (e.g., memories of hostprocessors, memories of reconfigurable processors, memories of NICdevices, memories on PCIe buses, memories on DDR channels, memories ofDIMMs, etc.).

In the implementation illustrated in FIG. 21, the local buses 125 a, 126a, 127 a, 125 n, 126 n, and 127 n and bus switches 124 a and 124 n thatoperatively couple reconfigurable processors on a same processing nodeto a host processor of the same processing node and to a NIC device or aSmartNIC device attached to the same processing node are PCIe buses 2132a, 2136 a, 2132 n, and 2136 n and PCIe switches (PEX) 2112 a, 2134 a,2112 n, and 2134 n, respectively. In other implementations, the PCIeprotocol can be replaced by or supplemented with other bus protocolssuch as Cache Coherent Interconnect for Accelerators (CCIX), ComputeExpress Link (CXL), and Open Coherent Accelerator Processor Interface(OpenCAPI).

Even though the message sequence chart 2100 begins at operation one,some preceding operations are omitted for the sake of clarity. Turningto the example illustrated in FIG. 18, some examples of the omittedoperations include the applications 1802 requesting execution, thecompiler 1812 compiling the applications 1802 and generating theconfiguration files 1822, the runtime processor 1832 allocating physicalresources, i.e., reconfigurable dataflow resources, for execution of theconfiguration files 1822, and the runtime processor 1832 loading theconfiguration files 1812 on the allocated reconfigurable dataflowresources. These omitted operations can be executed on any hostprocessor or any reconfigurable processor in the data center 100.

Continuing with the example illustrated in FIG. 18, consider that thevirtual dataflow resources 1824 and the virtual buffers 1824 c areallocated reconfigurable dataflow resources of the processing node 1 andthe processing node n in the data center 100. The first virtualprocessing node 1824 a is allocated the processing node 1 (hereinafter“a first processing node”). The first virtual reconfigurable processor1824 a 1 is allocated reconfigurable processor N (RP N) on theprocessing node 1 (hereinafter “a first reconfigurable processor”). Thesecond virtual processing node 1824 b is allocated the processing node n(hereinafter “a second processing node”). The second virtualreconfigurable processor 1824 b 1 is allocated reconfigurable processorN (RP N) on the processing node n (hereinafter “a second reconfigurableprocessor”). The first virtual SmartNIC buffers 1824 c 1 are allocatedthe sender buffers 2176 a and the receiver buffers 2178 a (hereinafter“a first plurality of buffers”). The second virtual SmartNIC buffers1824 c 2 are allocated the sender buffers 2176 n and the receiverbuffers 2178 n (hereinafter “a second plurality of buffers”).

The first plurality of buffers includes a first set of sender buffers2176 a configured to receive data from the first reconfigurableprocessor and provide the data to a second set of receiver buffers 2178n in the second plurality of buffers. The second set of receiver buffers2178 n are configured to provide the data to the second reconfigurableprocessor. The second plurality of buffers includes a second set ofsender buffers 2176 n configured to receive data from the secondreconfigurable processor and provide the data to a first set of receiverbuffers 2178 a in the first plurality of buffers. The first set ofreceiver buffers 2178 a are configured to provide the data to the firstreconfigurable processor.

The runtime processor 1832 is configured to configure the first SmartNIC132 a with a routing table that specifies the first reconfigurableprocessor as a local reconfigurable processor, and the secondreconfigurable processor as a destination reconfigurable processor. Theruntime processor 1832 is configured to configure the second SmartNIC132 n with a routing table that specifies the second reconfigurableprocessor as a local reconfigurable processor, and the firstreconfigurable processor as a destination reconfigurable processor.

In particular, FIG. 21 shows one implementation of how the runtimeprocessor 1832 executes the configuration files 1822 on the firstprocessing node (processing node 1) and the second processing node(processing node n). In one implementation, the execution includesstreaming data (e.g., configuration data (e.g., bit stream) andapplication data (weights, coefficients, vectors, tensors, control data(e.g., control tokens), etc.) for the configuration files 1822 thatdefine the applications 1802 between the first reconfigurable processorand the second reconfigurable processor using one or more buffers in thefirst plurality of buffers and one or more buffers in the secondplurality of buffers, thereby the streaming bypassing the first hostprocessor 102 a and the second host processor 102 n (as indicated by thedotted lines in FIG. 21). Accordingly, in some implementations, themessage sequence chart 2100 can be executed without using hostprocessors (e.g., as the host-less implementations discussed withrespect to FIG. 20). This saves latency and improves throughput, andalso does not require any processing time needed on the first and secondhost processors 102 a and 102 n (e.g., for processing by theirrespective operating systems).

In some implementations, the execution includes streaming input data forthe applications 1802 from the first reconfigurable processor to thesecond reconfigurable processor. In some implementations, one or more ofthe sender buffers in the first set of sender buffers 2176 a areconfigured to receive the input data from the first reconfigurableprocessor (operation one) and provide the input data to one or morereceiver buffers in the second set of receiver buffers 2178 n (operationtwo).

The first reconfigurable processor is configured to push the input datato the first SmartNIC 132 a (e.g., via the PCIe Endpoint Port (EP) 2146a) (operation one). In some implementations, operation one isaccomplished by an address generator of the first reconfigurableprocessor (e.g., Address Generation and Coalescing Units) (AGCU))writing the input data to physical memory addresses mapped to the senderbuffers in the first set of sender buffers 2176 a (e.g., via a hardwarewrite (HWRITE) command). In one implementation, the first SmartNIC 132 ais configured to write the input data, after encapsulation, into thesender buffers in the first set of sender buffers 2176 a. In oneimplementation, the first SmartNIC 132 a is configured to update tailpointers of the sender buffers in the first set of sender buffers 2176 ain response to the writing of the input data. In one implementation, thefirst SmartNIC 132 a is configured to process the input data as payload2156 a, apply encapsulation, store it in caches 2186 a, and stream it tothe second SmartNIC 132 n over the network fabric 136 (e.g., via the MACport 2196 a).

One skilled in the art will appreciate that operations one and sixcomprise streaming network packets between the first reconfigurableprocessor and the first SmartNIC 132 a over the local buses PCIe 2132 aand 2136 a using a protocol like Transaction Layer Packet (TLP) (e.g.,2120 a, 2128 a). One skilled in the art will also appreciate thatoperation two comprises streaming network packets from the firstSmartNIC 132 a to the second SmartNIC 132 n over the network fabric 136(e.g., Ethernet, InfiniBand (IB)) using protocols like RDMA overConverged Ethernet (RoCE), TCP, User Datagram Protocol (UDP), and QuickUDP Internet Connections (QUIC) (e.g., 2198 a, 2198 n).

The receiver buffers in the second set of receiver buffers 2178 n areconfigured to provide the input data to the second reconfigurableprocessor (operation three). In some implementations, operation three isaccomplished by an address generator of the second reconfigurableprocessor (e.g., Address Generation and Coalescing Units) (AGCU))reading the input data from physical memory addresses mapped to thereceiver buffers in the second set of receiver buffers 2178 n (e.g., viaa hardware read (HWREAD) command). In one implementation, the firstSmartNIC 132 a is configured to send the input data to the secondSmartNIC 132 n in response to the updated tail pointers. In oneimplementation, the second SmartNIC 132 n is configured to write theinput data, after decapsulation, into the receiver buffers in the secondset of receiver buffers 2178 n. In one implementation, the secondSmartNIC 132 n is configured to update tail pointers of the receiverbuffers in the second set of receiver buffers 2178 n in response to thewriting of the input data. The second reconfigurable processor isconfigured to pull the input data from the second SmartNIC 132 n (e.g.,via the PCIe Endpoint Port (EP) 2146 n) by reading the input data fromthe receiver buffers in the second set of receiver buffers 2178 n inresponse to the updated tail pointers.

In some implementations, the execution includes streaming output datafor the applications 1802 from the second reconfigurable processor tothe first reconfigurable processor. The output data is generated as aresult of processing the input data (e.g., processing of the input databy the second reconfigurable processor). In some implementations, one ormore of the sender buffers in the second set of sender buffers 2176 nare configured to receive the output data from the second reconfigurableprocessor (operation four) and provide the output data to one or morereceiver buffers in the first set of receiver buffers 2178 a (operationfive).

The second reconfigurable processor is configured to push the outputdata to the second SmartNIC 132 n (e.g., via the PCIe Endpoint Port (EP)2146 n) (operation four). In some implementations, operation four isaccomplished by an address generator of the second reconfigurableprocessor (e.g., Address Generation and Coalescing Units) (AGCU))writing the output data to physical memory addresses mapped to thesender buffers in the second set of sender buffers 2176 n (e.g., via ahardware write (HWRITE) command). In one implementation, the secondSmartNIC 132 n is configured to write the output data, afterencapsulation, into the sender buffers in the second set of senderbuffers 2176 n. In one implementation, the second SmartNIC 132 n isconfigured to update tail pointers of the sender buffers in the secondset of sender buffers 2176 n in response to the writing of the outputdata. In one implementation, the second SmartNIC 132 n is configured toprocess the output data as payload 2156 n, apply encapsulation, store itin caches 2186 n, and stream it to the first SmartNIC 132 a over thenetwork fabric 136 (e.g., via the MAC port 2196 n).

One skilled in the art will appreciate that operations three and fourcomprise streaming network packets between the second reconfigurableprocessor to the second SmartNIC 132 n over the local buses PCIe 2132 nand 2136 n using a protocol like Transaction Layer Packet (TLP) (e.g.,2120 n, 2128 n). One skilled in the art will also appreciate thatoperation five comprises streaming network packets from the secondSmartNIC 132 n to the first SmartNIC 132 a over the network fabric 136(e.g., Ethernet, InfiniBand (IB)) using protocols like RDMA overConverged Ethernet (RoCE), TCP, User Datagram Protocol (UDP), and QuickUDP Internet Connections (QUIC) (e.g., 2198 a, 2198 n).

The receiver buffers in the first set of receiver buffers 2178 a areconfigured to provide the output data to the first reconfigurableprocessor (operation six). In some implementations, operation six isaccomplished by an address generator of the first reconfigurableprocessor (e.g., Address Generation and Coalescing Units) (AGCU))reading the output data from physical memory addresses mapped to thereceiver buffers in the first set of receiver buffers 2178 a (e.g., viaa hardware read (HWREAD) command). In one implementation, the secondSmartNIC 132 n is configured to send the output data to the firstSmartNIC 132 a in response to the updated tail pointers. In oneimplementation, the first SmartNIC 132 a is configured to write theoutput data, after decapsulation, into the receiver buffers in the firstset of receiver buffers 2178 a. In one implementation, the firstSmartNIC 132 a is configured to update tail pointers of the receiverbuffers in the first set of receiver buffers 2178 a in response to thewriting of the output data. The first reconfigurable processor isconfigured to pull the output data from the first SmartNIC 132 a (e.g.,via the PCIe Endpoint Port (EP) 2146 a) by reading the output data fromthe receiver buffers in the first set of receiver buffers 2178 a inresponse to the updated tail pointers.

In some implementations, the first reconfigurable processor notifies thesecond reconfigurable processor of remote invocations using one or moreremote procedure calls. In one implementation, the first reconfigurableprocessor uses the sender buffers in the first set of sender buffers2176 a and the receiver buffers in the second set of receiver buffers2178 n to send, over the network fabric 136, one or more argument valuesto the second reconfigurable processor for execution of the remoteprocedure calls (similar to operation 2 in FIG. 21).

In some implementations, the second reconfigurable processor notifiesthe first reconfigurable processor of remote invocations using one ormore remote procedure calls. In one implementation, the secondreconfigurable processor uses the sender buffers in the second set ofsender buffers 2176 n and the receiver buffers in the first set ofreceiver buffers 2178 a to send, over the network fabric 136, one ormore argument values to the first reconfigurable processor for executionof the remote procedure calls (similar to operation 5 in FIG. 21).

FIG. 22 is a message sequence chart 2200 illustrating anotherimplementation of buffer-based inter-node streaming of configurationdata (e.g., bit stream) over the network fabric 136. In particular, FIG.22 shows another implementation of how the runtime processor 1832executes the configuration files 1822 on the first processing node(processing node 1) and the second processing node (processing node n).In one implementation, the execution includes streaming data (e.g.,configuration data (e.g., bit stream) and application data (weights,coefficients, vectors, tensors, control data (e.g., control tokens),etc.) for the configuration files 1822 that define the applications 1802between the first reconfigurable processor and the second host processor102 n using one or more buffers in the first plurality of buffers andone or more buffers in the second plurality of buffers, thereby thestreaming bypassing the first host processor 102 a (as indicated by thedotted lines in FIG. 22). This saves latency and improves throughput,and also does not require any processing time needed on the first hostprocessor 102 a (e.g., for processing by its operating system).

In some implementations, the execution includes streaming input data forthe applications 1802 from the first reconfigurable processor to thesecond host processor 102 n. In some implementations, one or more of thesender buffers in the first set of sender buffers 2176 a are configuredto receive the input data from the first reconfigurable processor(operation one) and provide the input data to one or more receiverbuffers in the second set of receiver buffers 2178 n (operation two).

The first reconfigurable processor is configured to push the input datato the first SmartNIC 132 a (e.g., via the PCIe endpoint port (EP) 2146a) (operation one). In some implementations, operation one isaccomplished by an address generator of the first reconfigurableprocessor (e.g., Address Generation and Coalescing Units) (AGCU))writing the input data to physical memory addresses mapped to the senderbuffers in the first set of sender buffers 2176 a (e.g., via a hardwarewrite (HWRITE) command). In one implementation, the first SmartNIC 132 ais configured to write the input data, after encapsulation, into thesender buffers in the first set of sender buffers 2176 a. In oneimplementation, the first SmartNIC 132 a is configured to update tailpointers of the sender buffers in the first set of sender buffers 2176 ain response to the writing of the input data. In one implementation, thefirst SmartNIC 132 a is configured to process the input data as payload2156 a, apply encapsulation, store it in caches 2186 a, and stream it tothe second SmartNIC 132 n over the network fabric 136 (e.g., via the MACport 2196 a).

One skilled in the art will appreciate that operations one and sixcomprise streaming network packets between the first reconfigurableprocessor and the first SmartNIC 132 a over the local buses PCIe 2132 aand 2136 a using a protocol like Transaction Layer Packet (TLP) (e.g.,2120 a, 2128 a). One skilled in the art will also appreciate thatoperation two comprises streaming network packets from the firstSmartNIC 132 a to the second SmartNIC 132 n over the network fabric 136(e.g., Ethernet, InfiniBand (IB)) using protocols like RDMA overConverged Ethernet (RoCE), TCP, User Datagram Protocol (UDP), and QuickUDP Internet Connections (QUIC) (e.g., 2198 a, 2198 n).

The receiver buffers in the second set of receiver buffers 2178 n areconfigured to provide the input data to the second host processor 102 n(operation three). In some implementations, operation three isaccomplished by an address generator of the second host processor 102 n(e.g., the second host processor reads DMAed data once the DMA operationis complete.) reading the input data from physical memory addressesmapped to the receiver buffers in the second set of receiver buffers2178 n (e.g., via a hardware read (HWREAD) command). In oneimplementation, the first SmartNIC 132 a is configured to send the inputdata to the second SmartNIC 132 n in response to the updated tailpointers. In one implementation, the second SmartNIC 132 n is configuredto write the input data, after decapsulation, into the receiver buffersin the second set of receiver buffers 2178 n. In one implementation, thesecond SmartNIC 132 n is configured to update tail pointers of thereceiver buffers in the second set of receiver buffers 2178 n inresponse to the writing of the input data. The second host processor 102n is configured to pull the input data from the second SmartNIC 132 n(e.g., via the PCIe Endpoint Port (EP) 2146 n) by reading the input datafrom the receiver buffers in the second set of receiver buffers 2178 nin response to the updated tail pointers. Generally SmartNIC would DMAthe payload into host 102 n memory 134 n, then notify the host via a DMAcompletion mechanism.

In some implementations, the execution includes streaming output datafor the applications 1802 from the second host processor 102 n to thefirst reconfigurable processor. The output data is generated as a resultof processing the input data (e.g., processing of the input data by thesecond host processor 102 n). In some implementations, one or more ofthe sender buffers in the second set of sender buffers 2176 n areconfigured to receive the output data from the second host processor 102n (operation four) and provide the output data to one or more receiverbuffers in the first set of receiver buffers 2178 a (operation five).

The second host processor 102 n is configured to push the output data tothe second SmartNIC 132 n (e.g., via the PCIe Endpoint Port (EP) 2146 n)(operation four). In some implementations, operation four isaccomplished by a DMA operation. In one implementation, the secondSmartNIC 132 n is configured to write the output data, afterencapsulation, into the sender buffers in the second set of senderbuffers 2176 n. In one implementation, the second SmartNIC 132 n isconfigured to update tail pointers of the sender buffers in the secondset of sender buffers 2176 n in response to the writing of the outputdata. In one implementation, the second SmartNIC 132 n is configured toprocess the output data as payload 2156 n, apply encapsulation, store itin caches 2186 n, and stream it to the first SmartNIC 132 a over thenetwork fabric 136 (e.g., via the MAC port 2196 n).

One skilled in the art will appreciate that operations three and fourcomprise streaming network packets between the second host processor 102n to the second SmartNIC 132 n over the local buses PCIe 2132 n and 2136n using a protocol like Transaction Layer Packet (TLP) (e.g., 2120 n,2128 n). One skilled in the art will also appreciate that operation fivecomprises streaming network packets from the second SmartNIC 132 n tothe first SmartNIC 132 a over the network fabric 136 (e.g., Ethernet,InfiniBand (IB)) using protocols like RDMA over Converged Ethernet(RoCE), TCP, User Datagram Protocol (UDP), and Quick UDP InternetConnections (QUIC) (e.g., 2198 a, 2198 n).

The receiver buffers in the first set of receiver buffers 2178 a areconfigured to provide the output data to the first reconfigurableprocessor (operation six). In some implementations, operation six isaccomplished by an address generator of the first reconfigurableprocessor (e.g., Address Generation and Coalescing Units) (AGCU))reading the output data from physical memory addresses mapped to thereceiver buffers in the first set of receiver buffers 2178 a (e.g., viaa hardware read (HWREAD) command). In one implementation, the secondSmartNIC 132 n is configured to send the output data to the firstSmartNIC 132 a in response to the updated tail pointers. In oneimplementation, the first SmartNIC 132 a is configured to write theoutput data, after decapsulation, into the receiver buffers in the firstset of receiver buffers 2178 a. In one implementation, the firstSmartNIC 132 a is configured to update tail pointers of the receiverbuffers in the first set of receiver buffers 2178 a in response to thewriting of the output data. The first reconfigurable processor isconfigured to pull the output data from the first SmartNIC 132 a (e.g.,via the PCIe Endpoint Port (EP) 2146 a) by reading the output data fromthe receiver buffers in the first set of receiver buffers 2178 a inresponse to the updated tail pointers.

In some implementations, the first reconfigurable processor notifies thesecond host processor 102 n of remote invocations using one or moreremote procedure calls. In one implementation, the first reconfigurableprocessor uses the sender buffers in the first set of sender buffers2176 a and the receiver buffers in the second set of receiver buffers2178 n to send, over the network fabric 136, one or more argument valuesto the second host processor 102 n for execution of the remote procedurecalls (similar to operation 2 in FIG. 22).

In some implementations, the second host processor 102 n notifies thefirst reconfigurable processor of remote invocations using one or moreremote procedure calls. In one implementation, the second host processor102 n uses the sender buffers in the second set of sender buffers 2176 nand the receiver buffers in the first set of receiver buffers 2178 a tosend, over the network fabric 136, one or more argument values to thefirst reconfigurable processor for execution of the remote procedurecalls (similar to operation 5 in FIG. 22).

In the synchronous mode of a remote procedure call using a first set ofbuffers, the technology disclosed allows a remote entity which executedthe remote procedure call to produce one or more result values and sendthem back to the remote caller using a distinct set of buffers. In oneimplementation, the two communicating entities may designate two bufferqueues, one in each direction. The caller will send the data by copyingit into a first buffer queue. The receiver will pull the data out of thefirst buffer queue, compute an operation, and then place the result in asecond buffer queue. The original caller will simply wait until thesecond buffer queue has data available and will be able to use theresult computed remotely as soon as it arrives over the second bufferqueue.

In other implementations of the technology disclosed, SmartNICs can bereplaced by NICs, which can be controlled by NIC DMAs or via the hostprocessors to implement the flows illustrated in FIGS. 21 and 22 (e.g.,updating the head and tail pointers of the buffers). For example, in theNIC implementations, operations two and five of FIGS. 21 and 22 areexecuted by the first and second host processors 102 a and 102 n byinitiating Remote DMA (RDMA) of the networking packets between the firstNIC 132 a and the second NIC 132 n, and updating the corresponding tailpointers of the buffers upon arrival of the network packets.

In some implementations, the SmartNICs and the NICs are embedded on-chipon the reconfigurable processors.

Model Parallelism

FIG. 23 illustrates one implementation of executing 2300 amodel/application in parallel using the disclosed buffer-basedinter-node streaming of configuration data (e.g., bit stream) over thenetwork fabric 136. This is referred to herein as “model parallelism.”

Application 2302 is a dataflow graph with a set of processing modules(e.g., processing modules 1 to 5). Examples of the processing modulesinclude neurons or layers of deep neural networks. The runtime processor1832 is configured to partition the set of processing modules into afirst subset of processing modules 2304 a and a second subset ofprocessing modules 2304 b. The runtime processor 1832 is configured toexecute configuration files 2322 a for the first subset of processingmodules 2304 a on the first reconfigurable processor (e.g., RP N fromthe RPs 142 a on the processing node 1). The runtime processor 1832 isconfigured to execute configuration files 2322 b for the second subsetof processing modules 2304 b on the second reconfigurable processor(e.g., RP N from the RPs 142 n on the processing node n).

Deep neural network training, implemented, for example, by StochasticGradient Descent (SGD) comprises a forward pass and a backward pass. Thebackward pass comprises a delta pass and a chain pass. The forward passpropagates activations in a forward direction. The delta pass propagatesdeltas in a backward direction. The chain pass calculates gradientsbased on the deltas as the deltas are generated in the delta pass.

The runtime processor 1832 is configured to use the first plurality ofbuffers 2176 a, 2178 a and the second plurality of buffers 2176 n, 2178n to stream data between the first subset of processing modules 2304 aand the second subset of processing modules 2304 b. The data includesfeature maps and/or activations generated during a forward pass, andloss gradients generated during a backward pass.

The operations one to six depicted in FIG. 23 are similar tocorresponding operations in FIG. 21.

Data Parallelism

FIG. 24 illustrates one implementation of executing 2400 multipleinstances of a model/application in parallel using the disclosedbuffer-based inter-node streaming of configuration data (e.g., bitstream) over the network fabric 136. This is referred to herein as “dataparallelism.” The runtime processor 1832 is configured to initialize afirst instance of the dataflow graph 2404 a and a second instance of thedataflow graph 2404 b.

The runtime processor 1832 is configured to execute configuration files2422 a for the first instance 2404 a of the dataflow graph on the firstreconfigurable processor (e.g., RP N from the RPs 142 a on theprocessing node 1). The runtime processor 1832 is configured to executeconfiguration files 2422 b for the second instance 2404 b of thedataflow graph on the second reconfigurable processor (e.g., RP N fromthe RPs 142 n on the processing node n).

The runtime processor 1832 is configured to use the first plurality ofbuffers 2176 a, 2178 a and the second plurality of buffers 2176 n, 2178n to stream data between the first instance of the dataflow graph andthe second instance of the dataflow graph. The data includes gradientsgenerated during the backward pass.

The operations one to six depicted in FIG. 24 are similar tocorresponding operations in FIG. 21.

Heterogeneous Reconfigurable Processors

FIG. 25 illustrates one implementation of executing 2500 configurationfiles on heterogeneous reconfigurable processors (e.g., RP 1 and RP 2 inFIG. 25). Examples of the heterogeneous reconfigurable processorsinclude Central Processing Units (CPUs), Graphics Processing Units(GPUs), Field Programmable Gate Arrays (FPGAs), Coarse-GrainedReconfigurable Architectures (CGRAs), Application-Specific IntegratedCircuits (ASICs), Application Specific Instruction-set Processor (ASIP),and Digital Signal Processors (DSPs).

The heterogeneous reconfigurable processors have different levels ofconfigurable granularity. The runtime processor 1832 is configured toreceive a set of configuration files (e.g., 1822) for an application(e.g., 1802). The runtime processor 1832 is configured to load andexecute a first subset of configuration files 2502 a in the set ofconfiguration files on a first reconfigurable processor (RP 1) in theheterogeneous reconfigurable processors. The first reconfigurableprocessor has a first configuration and/or a first level of configurablegranularity. The runtime processor 1832 is configured to load andexecute a second subset of configuration files 2502 b in the set ofconfiguration files on a second reconfigurable processor (RP 2) in theheterogeneous reconfigurable processors. The second reconfigurableprocessor has a second configuration and/or a second level ofconfigurable granularity that is different from the first configurationand/or the first level of configurable granularity.

The first level of configurable granularity is a bit-level configurablegranularity, and the first reconfigurable processor is aField-Programmable Gate Array (FPGA). The second level of configurablegranularity is a word-level configurable granularity, and the secondreconfigurable processor is a Coarse-Grained Reconfigurable Architecture(CGRA).

The first configuration is a bit-level configurable granularity, and thefirst reconfigurable processor is a Field-Programmable Gate Array(FPGA). The second configuration is a word-level configurablegranularity, and the second reconfigurable processor is a Coarse-GrainedReconfigurable Architecture (CGRA). The first configuration is agate-level reconfigurability, and the first reconfigurable processor isthe FPGA. The second configuration is a register transfer-levelreconfigurability, and the second reconfigurable processor is the CGRA.The first configuration uses bit-wise Look-Up Tables (LUTs) andswitches, and the first reconfigurable processor is the FPGA. The secondconfiguration uses word-wide Issue Slots (ISs)/Arithmetic Logic Units(ALUs)/Functional Units (FUs)/Processing Elements (PEs), Register Files(RFs), and interconnections, and the second reconfigurable processor isthe CGRA. A number of the ISs used by the second reconfigurableprocessor is fewer than a number of the LUTs used by the firstreconfigurable processor. A number of bits required to configure thesecond reconfigurable processor is orders of magnitude smaller than anumber of bits required to configure the first reconfigurable processor.

On-Chip NIC/SmartNIC

FIG. 26 illustrates one implementation of executing 2600 configurationfiles using NIC or SmartNIC devices that are embedded on thereconfigurable processors.

A first reconfigurable processor (e.g., RP N from the RPs 142 a on theprocessing node 1) has a first Network Interface Controller (NIC), andthe first NIC has a first plurality of buffers 2176 a, 2178 a. A secondreconfigurable processor (e.g., RP N from the RPs 142 n on theprocessing node n) has a second NIC, and the second NIC has a secondplurality of buffers 2176 n, 2178 n. The runtime processor 1832 isconfigured to execute the configuration files 1812 for the applications1802 using the first reconfigurable processor and the secondreconfigurable processor. The execution includes streaming data (e.g.,configuration data (e.g., bit stream) and application data (weights,coefficients, vectors, tensors, control data (e.g., control tokens),etc.) for the configuration files 1822 that define the applications 1802between the first reconfigurable processor and the second reconfigurableprocessor using the first plurality of buffers of the first NIC and thesecond plurality of buffers of the second NIC.

The operations one to six depicted in FIG. 26 are similar tocorresponding operations in FIG. 21.

Example Reconfigurable Processor

FIG. 27 is a diagram illustrating a system 2700 including a host 2720, amemory 2740, and an example reconfigurable data processor 2710 in whicha computation unit as described herein is deployed by hardware or byconfiguration of reconfigurable components and configured with thevirtualization logic 2797. As shown in the example of FIG. 27, thereconfigurable data processor 2710 includes an array 2790 ofconfigurable units and a configuration load/unload controller 2795.

The virtualization logic 2797 can include resources that support orenable simultaneous execution of multiple, unrelated application graphs(or related ones) in an array of configurable units on one die or onemultichip module. In the illustration, a first application graph isimplemented in virtual machine VM1 in a particular set 2798 ofconfigurable units, and a second application graph is implemented invirtual machine VM2 in another set 2799 of configurable units.

Configurable units in an array 2790 of configurable units are furtherdescribed in reference to FIGS. 30 and 31 and configured with thevirtualization logic 2797. Configurable units can include, or can haveunits configured to implement, a computation unit or computation units,as described herein.

The reconfigurable data processor 2710 includes an external I/Ointerface 2730 connected to the host 2720 by line 2725, and an externalI/O interface 2750 connected to the memory 2740 by line 2745. The I/Ointerfaces 2730, 2750 connect via a bus system 2715 to the array 2790 ofconfigurable units and to the configuration load/unload controller 2795.The bus system 2715 may have a bus width of carrying one chunk of data,which can be for this example one hundred and twenty-eight bits(references to one hundred and twenty-eight bits throughout can beconsidered as an example chunk size more generally).

To configure configurable units in the array 2790 of configurable unitswith a configuration file, the host 2720 can send the configuration fileto the memory 2740 via the I/O interface 2730, the bus system 2715, andthe I/O interface 2750 in the reconfigurable data processor 2710. Theconfiguration file can be loaded in many ways, as suits a particulararchitecture, including in data paths outside the configurable processor2710. The configuration file can be retrieved from the memory 2740 viathe memory I/O interface 2750. Chunks of the configuration file can thenbe sent in a distribution sequence to configurable units in the array2790 of configurable units in the reconfigurable data processor 2710.

An external clock generator 2770 or other clock line sources can providea clock line 2775 or clock lines to elements in the reconfigurable dataprocessor 2710, including the array 2790 of configurable units, and thebus system 2715, and the external data I/O interfaces. The bus system2715 can communicate data at a processor clock rate via a clock line2775 or clock lines.

FIG. 28 is a simplified block diagram 2800 of components of a CGRA(Coarse-Grained Reconfigurable Architecture) processor. In this example,the CGRA processor has two tiles (Tile1, Tile2). The tile comprises anarray of configurable units connected to a bus system, including arraylevel networks in this example. An array of configurable units (e.g.,2790, FIG. 27) in the tile includes computation units in hardware or byconfiguration of reconfigurable components, which are configured withthe virtualization logic 2797. The bus system includes a top-levelnetwork connecting the tiles to external I/O interface 2805 (or anynumber of interfaces). In other embodiments, different bus systemconfigurations may be utilized. The configurable units in each tile arenodes on the array level network in this embodiment.

Each of the tiles has four AGCUs (Address Generation and CoalescingUnits) (e.g., MAGCU1, AGCU9, AGCU13, AGCU14). The AGCUs are nodes on thetop-level network and nodes on the array level networks and includeresources for routing data among nodes on the top-level network andnodes on the array level network in each tile.

Nodes on the top-level network in this example include one or moreexternal I/Os, including interface 2805. The interfaces to externaldevices include resources for routing data among nodes on the top-levelnetwork and external devices, such as high-capacity memory, hostprocessors, other CGRA processors, FPGA devices and so on, that areconnected to the interfaces.

One of the AGCUs in a tile is configured in this example to be a masterAGCU, which includes an array configuration load/unload controller forthe tile. In other embodiments, more than one array configurationload/unload controller can be implemented, and one array configurationload/unload controller may be implemented by logic distributed amongmore than one AGCU.

The MAGCU1 includes a configuration load/unload controller for Tile1,and MAGCU2 includes a configuration load/unload controller for Tile2. Inother embodiments, a configuration load/unload controller can bedesigned for loading and unloading configuration of more than one tile.In other embodiments, more than one configuration controller can bedesigned for configuration of a single tile. Also, the configurationload/unload controller can be implemented in other portions of thesystem, including as a stand-alone node on the top-level network and thearray level network or networks.

The top-level network is constructed using top-level switches (2811,2813, 2814, and 2816) connecting to each other as well as to other nodeson the top-level network, including the AGCUs, and I/O interface 2805.The top-level network includes links (e.g., L11, L9, L21, L22)connecting the top-level switches. Data travels in packets between thetop-level switches on the links, and from the switches to the nodes onthe network connected to the switches. For example, top-level switches2811 and 2812 are connected by a link L14, top-level switches 2814 and2815 are connected by a link L9, top-level switches 2811 and 2814 areconnected by a link L13, and top-level switches 2812 and 2813 areconnected by a link L21. The links can include one or more buses andsupporting control lines, including for example a chunk-wide bus (vectorbus). For example, the top-level network can include data, request andresponse channels operable in coordination for transfer of data in amanner analogous to an AXI compatible protocol. See, AMBA® AXI and ACEProtocol Specification, ARM.

Top-level switches can be connected to AGCUs. For example, top-levelswitches 2811, 2812, 2814, and 2815 are connected to MAGCU1, AGCU9,AGCU13 and AGCU14 in the tile Tile1, respectively. Top-level switches2812, 2813, 2815, and 2816 are connected to MAGCU2, AGCU22, AGCU23 andAGCU24 in the tile Tile2, respectively.

Top-level switches can be connected to one or more external I/Ointerfaces (e.g., interface 2805).

FIG. 29A is a simplified diagram of a tile and an array level networkusable in the configuration of FIG. 28, where the configurable units inthe array are nodes on the array level network and are configurable toimplement the virtualization logic 2797.

In this example, the array of configurable units 2900 includes aplurality of types of configurable units, which are configured with thevirtualization logic 2797. The types of configurable units in thisexample, include Pattern Compute Units (PCUs), Pattern Memory Units(PMUs), Switch units (S), and Address Generation and Coalescing Units(each including two address generators AG and a shared CU). For anexample of the functions of these types of configurable units, see,Prabhakar et al., “Plasticine: A Reconfigurable Architecture ForParallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada,which is incorporated by reference as if fully set forth herein. In thisexample, the PCUs (e.g., 2942) and PMUs (e.g., 2943) in the array ofconfigurable units 2900 can include resources configurable forembodiment of a computation unit, an example configuration of which isdescribed herein. Each of these configurable units contains aconfiguration store comprising a set of registers or flip-flops thatrepresent either the setup or the sequence to run a program, and caninclude the number of nested loops, the limits of each loop iterator,the routes and/or instructions to be executed for each stage includingstages, the source of the operands, and the network parameters for theinput and output interfaces. The configuration file can include entriesof Look-Up Tables as described herein.

Additionally, each of these configurable units contains a configurationstore comprising a set of registers or flip-flops that store statususable to track progress in nested loops or otherwise. A configurationfile in the configuration store contains a bit-stream representing theinitial configuration, or starting state, of each of the components thatexecute the program. This bit-stream is referred to as a bit file.Program load is the process of setting up the configuration stores inthe array of configurable units based on the contents of the bit file toallow the components to execute a program (i.e., a machine), includingprograms that utilize the virtualization logic 2797. Program Load mayalso require the load of all PMU memories.

The array level network includes links interconnecting configurableunits in the array. The links in the array level network include one ormore and, in this case, three kinds of physical buses: a chunk-levelvector bus (e.g., one hundred and twenty-eight bits of data), aword-level scalar bus (e.g., thirty-two bits of data), and a multiplebit-level control bus. For instance, interconnect 2921 between switchunits 2911 and 2912 includes a vector bus interconnect with a vector buswidth of one hundred and twenty-eight bits, a scalar bus interconnectwith a scalar bus width of thirty-two bits, and a control businterconnect.

The three kinds of physical buses differ in the granularity of databeing transferred. In one embodiment, the vector bus can carry a chunkthat includes sixteen-bytes (=one hundred and twenty-eight bits) of dataas its payload. The scalar bus can have a thirty-two-bit payload andcarry scalar operands or control information. In some machinesimplemented using this system, data can be represented using floatingpoint data formats, including standard or non-standard formats. Exampleformats include FP32 and BF16, among others. It can be understood thatthe number of data values carried on the scalar and vector buses is afunction of the encoding format of the data values, with FP32 utilizingthirty-two bits per value and BF16 using sixteen bits per value.

The control bus can carry control handshakes such as tokens and otherlines. The vector and scalar buses can be packet switched, includingheaders that indicate a destination of each packet and other informationsuch as sequence numbers that can be used to reassemble a file when thepackets are received out of order. Each packet header can contain adestination identifier that identifies the geographical coordinates ofthe destination switch unit (e.g., the row and column in the array), andan interface identifier that identifies the interface on the destinationswitch (e.g., North, South, East, West, etc.) used to reach thedestination unit. The control network can be circuit switched based ontiming circuits in the device, for example. The configurationload/unload controller can generate a header for each chunk ofconfiguration data (e.g., bit stream) of one hundred and twenty-eightbits. The header is transmitted on a header bus to each configurableunit in the array of configurable unit.

In one example, a chunk of data of one hundred and twenty-eight bits istransmitted on the vector bus that provides the chunk as vector inputsto a configurable unit. The vector bus can include one hundred andtwenty-eight payload lines, and a set of header lines. The header caninclude a sequence ID for each chunk, which can include:

-   -   A bit to indicate if the chunk is scratchpad memory or        configuration store data.    -   Bits that form a chunk number.    -   Bits that indicate a column identifier.    -   Bits that indicate a row identifier.    -   Bits that indicate a component identifier.

For a load operation, the configuration load controller can send thenumber N of chunks to a configurable unit in order from N−1 to 0. If,for example, N=6, the chunks are sent out in most-significant-bit-firstorder of Chunk 5->Chunk 4->Chunk 3->Chunk 2->Chunk 1->Chunk 0. (Notethat this most-significant-bit-first order results in Chunk 5 beingdistributed in round 0 of the distribution sequence from the arrayconfiguration load controller.) For an unload operation, theconfiguration unload controller can write the unload data out of orderto the memory. For both load and unload operations, the shifting in theconfiguration serial chains in a configuration data (e.g., bit stream)store in a configurable unit is from LSB (Least-Significant-Bit) to MSB(Most-Significant-Bit), or MSB out first.

FIG. 29B illustrates an example switch unit connecting elements in anarray level network. As shown in the example of FIG. 29B, a switch unitcan have eight interfaces. The North, South, East and West interfaces ofa switch unit are used for connections between switch units. TheNortheast, Southeast, Northwest and Southwest interfaces of a switchunit are each used to make connections to PCU or PMU instances. A set oftwo switch units in each tile quadrant have connections to an AddressGeneration and Coalescing Unit (AGCU) that include multiple AddressGeneration (AG) units and a Coalescing Unit (CU) connected to themultiple address generation units. The Coalescing Unit (CU) arbitratesbetween the AGs and processes memory requests. Each of the eightinterfaces of a switch unit can include a vector interface, a scalarinterface, and a control interface to communicate with the vectornetwork, the scalar network, and the control network.

During execution of a machine after configuration, data can be sent viaone or more unit switches and one or more links between the unitswitches to the configurable units using the vector bus and vectorinterface(s) of the one or more switch units on the array level network.

In embodiments described herein, a configuration file or bit file,before configuration of the tile, can be sent from the configurationload controller using the same vector bus, via one or more unit switchesand one or more links between the unit switches to the configurable unitusing the vector bus and vector interface(s) of the one or more switchunits on the array level network. For instance, a chunk of configurationdata (e.g., bit stream) in a unit file particular to a configurable unitPMU 2941 can be sent from the configuration load/unload controller 2901to the PMU 2941, via a link 2920 between the configuration load/unloadcontroller 2901 and the West (W) vector interface of the switch unit2911, the switch unit 2911, and a link 2931 between the Southeast (SE)vector interface of the switch unit 2911 and the PMU 2941.

In this example, one of the AGCUs is configured to be a master AGCU,which includes a configuration load/unload controller (e.g., 2901). Themaster AGCU implements a register through which the host (2720, FIG. 27)can send commands via the bus system to the master AGCU. The master AGCUcontrols operations on an array of configurable units in a tile andimplements a program control state machine to track the state of thetile based on the commands it receives from the host through writes tothe register. For every state transition, the master AGCU issuescommands to all components on the tile over a daisy-chained command bus(FIG. 30). The commands include a program reset command to resetconfigurable units in an array of configurable units in a tile, and aprogram load command to load a configuration file to the configurableunits.

The configuration load controller in the master AGCU is responsible forreading the configuration file from the memory and sending theconfiguration data (e.g., bit stream) to every configurable unit of thetile. The master AGCU can read the configuration file from the memory atpreferably the maximum throughput of the top-level network. The dataread from memory are transmitted by the master AGCU over the vectorinterface on the array level network to the corresponding configurableunit according to a distribution sequence described herein.

In one embodiment, in a way that can reduce the wiring requirementswithin a configurable unit, configuration and status registers holdingunit files to be loaded in a configuration load process, or unloaded ina configuration unload process, in a component are connected in a serialchain and can be loaded through a process of shifting bits through theserial chain. In some embodiments, there may be more than one serialchain arranged in parallel or in series. When a configurable unitreceives, for example, one hundred and twenty-eight bits ofconfiguration data (e.g., bit stream) from the master AGCU in one buscycle, the configurable unit shifts this data through its serial chainat the rate of one bit per cycle, where shifter cycles can run at thesame rate as the bus cycle. It will take one hundred and twenty-eightshifter cycles for a configurable unit to load one hundred andtwenty-eight configuration bits with the one hundred and twenty-eightbits of data received over the vector interface. The one hundred andtwenty-eight bits of configuration data (e.g., bit stream) are referredto as a chunk. A configurable unit can require multiple chunks of datato load all its configuration bits.

The configurable units interface with the memory through multiple memoryinterfaces (2750, FIG. 27). Each of the memory interfaces can beaccessed using several AGCUs. Each AGCU contains a reconfigurable scalardata path to generate requests for the off-chip memory. Each AGCUcontains FIFOs (First-In, First-Out buffers for organizing data) tobuffer outgoing commands, data, and incoming responses from the off-chipmemory.

FIG. 30 is a block diagram illustrating an example configurable unit3000, such as a Pattern Compute Unit (PCU), which is configured with thevirtualization logic 2797. A configurable unit can interface with thescalar, vector, and control buses, in this example using threecorresponding sets of inputs and outputs (IO): scalar inputs/outputs,vector inputs/outputs, and control inputs/outputs. Scalar IOs can beused to communicate single words of data (e.g., thirty-two bits). VectorIOs can be used to communicate chunks of data (e.g., one hundred andtwenty-eight bits), in cases such as receiving configuration data (e.g.,bit stream) in a unit configuration load process and transmitting andreceiving data during operation after configuration across a longpipeline between multiple PCUs. Control IOs can be used to communicatesignals on control lines such as the start or end of execution of aconfigurable unit. Control inputs are received by control block 3090,and control outputs are provided by the control block 3090.

Each vector input is buffered in this example using a vector FIFO in avector FIFO block 3060 which can include one or more vector FIFOs.Likewise, in this example, each scalar input is buffered using a scalarFIFO 3070. Using input FIFOs decouples timing between data producers andconsumers and simplifies inter-configurable-unit control logic by makingit robust to input delay mismatches.

A configurable unit includes multiple reconfigurable data paths in block3080. A data path in a configurable unit can be organized as amulti-stage (Stage 1 . . . Stage N), reconfigurable SIMD (SingleInstruction, Multiple Data) pipeline. The chunks of data pushed into theconfiguration serial chain in a configurable unit include configurationdata (e.g., bit stream) for each stage of each data path in theconfigurable unit. The configuration serial chain in the configurationdata (e.g., bit stream) store 3020 is connected to the multiple datapaths in block 3080 via lines 3021.

A configurable data path organized as a multi-stage pipeline can includemultiple functional units (e.g., 3081, 3082, 3083, 3084, 3085, 3086) atrespective stages. A computation unit or parts of a computation unit canbe implemented in multiple functional units at respective stages in amulti-stage pipeline or in multiple multi-stage pipelines. A circuitincluding the virtualization logic 2797 can be implemented in multiplefunctional units and multiple memory units. Input registers infunctional units can register inputs from scalar FIFOs 3070 or VectorFIFOs 3060 or from previous stages in a multi-stage pipeline. Afunctional unit at a stage in a multi-stage pipeline can execute afunction, e.g., logical shift, an arithmetic function, comparison, alogical operation, etc., and generate an output.

Configurable units in the array of configurable units includeconfiguration data (e.g., bit stream) stores 3020 (e.g., serial chains)to store unit files comprising a plurality of chunks (or sub-files ofother sizes) of configuration data (e.g., bit stream) particular to thecorresponding configurable units. Configurable units in the array ofconfigurable units each include unit configuration load logic 3040connected to the configuration data (e.g., bit stream) store 3020 vialine 3022, to execute a unit configuration load process. The unitconfiguration load process includes receiving, via the bus system (e.g.,the vector inputs), chunks of a unit file particular to the configurableunit and loading the received chunks into the configuration data (e.g.,bit stream) store 3020 of the configurable unit. The unit file loadedinto the configuration data (e.g., bit stream) store 3020 can includeconfiguration data (e.g., bit stream), including opcodes and routingconfiguration, for circuits (e.g., module) implementing thevirtualization logic 2797 in multiple functional units and multiplememory units, as described herein.

The configuration data (e.g., bit stream) stores in configurable unitsin the plurality of configurable units in this example comprise serialchains of latches, where the latches store bits that controlconfiguration of the resources in the configurable unit. A serial chainin a configuration data (e.g., bit stream) store can include a shiftregister chain for configuration data (e.g., bit stream) and a secondshift register chain for state information and counter values connectedin series.

Input configuration data (e.g., bit stream) 3010 can be provided to avector FIFO as vector inputs, and then be transferred to theconfiguration data (e.g., bit stream) store 3020. Output configurationdata (e.g., bit stream) 3030 can be unloaded from the configuration data(e.g., bit stream) store 3020 using the vector outputs.

The CGRA uses a daisy-chained completion bus to indicate when aload/unload command has been completed. The master AGCU transmits theprogram load and unload commands to configurable units in the array ofconfigurable units over a daisy-chained command bus. As shown in theexample of FIG. 30, a control block 3090, a daisy-chained completion bus3091 and a daisy-chained command bus 3092 are connected to daisy-chainlogic 3093, which communicates with the unit configuration load logic3040. The daisy-chain logic 3093 can include load complete status logic,as described below. The daisy-chained completion bus is furtherdescribed below. Other topologies for the command and completion busesare clearly possible but not described here.

FIG. 31 is a block diagram illustrating an example configurable unit3100, such as a Pattern Memory Unit (PMU), which is configured with thevirtualization logic 2797 (i.e., ready-to-read credit counters, writecredit counters, and flow control logic for operating them). A PMU cancontain scratchpad memory 3130 coupled with a reconfigurable scalar datapath 3120 intended for address calculation (RA, WA) and control (WE, RE)of the scratchpad memory 3130, along with the bus interfaces used in thePCU.

The bus interfaces can include scalar inputs, vector inputs, scalaroutputs and vector outputs, usable to provide write data WD. The datapath can be organized as a multi-stage reconfigurable pipeline,including stages of functional units FUs and associated pipelineregisters PRs that register inputs and outputs of the functional units.PMUs can be used to store distributed on-chip memory throughout thearray of reconfigurable units.

A scratchpad is built with multiple SRAM banks (e.g., 3131, 3132, 3133,3134). Banking and buffering logic 3135 for the SRAM banks in thescratchpad can be configured to operate in several banking modes tosupport various access patterns. A computation unit as described hereincan include a Look-Up Table stored in the scratchpad memory 3130, from aconfiguration file or from other sources. In a computation unit asdescribed herein, the scalar data path 3120 can translate a section of araw input value I for addressing Look-Up Tables implementing a functionf(I), into the addressing format utilized by the SRAM scratchpad memory3130, adding appropriate offsets and so on, to read the entries of theLook-Up Table stored in the scratchpad memory 3130 using the sections ofthe input value I. Each PMU can include write address calculation logicand read address calculation logic that provide write address WA, writeenable WE, read address RA and read enable RE to the banking bufferinglogic 3135. Based on the state of the local FIFOs 3111 and 3112 andexternal control inputs, the control block 3115 can be configured totrigger the write address computation, read address computation, orboth, by enabling the appropriate counters 3116. A programmable counterchain 3116 (Control Inputs, Control Outputs) and control block 3115 cantrigger PMU execution.

This is one simplified example of a configuration of a configurableprocessor for implementing a computation unit as described herein. Theconfigurable processor can be configured in other ways to implement acomputation unit. Other types of configurable processors can implementthe computation unit in other ways. Also, the computation unit can beimplemented using dedicated logic in some examples, or a combination ofdedicated logic and instruction-controlled processors.

Other Implementations

In various implementations of the technology disclosed, when two or morereconfigurable processors collaboratively execute an application, thetwo or more reconfigurable processors are independently and separatelyconfigured (e.g., by the runtime processor) with a same set ofconfiguration files. In one implementation, when a first reconfigurableprocessor, configured with a given set of configuration files, beginsexecuting configuration files in the given set of configuration filesand/or functions therefor and/or data therefor, and requires a secondreconfigurable processor, also configured with the given set ofconfiguration files, to execute certain configuration files in the givenset of configuration files and/or functions therefor and/or datatherefor, then the second reconfigurable processor waits for a signalfrom the first reconfigurable processor. Examples of the signal includea control signal that indicates a breakpoint/checkpoint after a quiescecondition, such as the one described in U.S. Non-provisional patentapplication Ser. No. 16/504,627, filed Jul. 8, 2019, entitled, “QUIESCERECONFIGURABLE DATA PROCESSOR,” (Attorney Docket No. SBNV 1008-1).

Then, after receiving the signal and corresponding application data andtensor state from the first reconfigurable processor, the secondreconfigurable processor begins execution of the certain configurationfiles and/or functions therefor and/or data therefor using its own copyof the given set of configuration files with which it is independentlyand separately configured. In some implementations, a checkpoint isgenerated at the first reconfigurable processor, the checkpoint istransferred to the second reconfigurable processor, and the secondreconfigurable processor loads the checkpoint and begins execution ofthe certain configuration files and/or functions therefor and/or datatherefor.

A first example of accelerated deep learning is using a deep learningaccelerator to train a neural network. A second example of accelerateddeep learning is using a deep learning accelerator to operate a trainedneural network to perform inferences. A third example of accelerateddeep learning is using a deep learning accelerator to train a neuralnetwork and subsequently perform inference with any one or more of thetrained neural network, information from same, and a variant of same.

Examples of neural networks include Fully Connected Neural Networks(FCNNs), Recurrent Neural Networks (RNNs), Convolutional Neural Networks(CNNs), Long Short-Term Memory (LSTM) networks, autoencoders, deepbelief networks, and Generative Adversarial Networks (GANs).

An example of training a neural network is determining one or moreweights associated with the neural network, such as by hardwareacceleration via a deep learning accelerator. An example of making aninference is using a trained neural network to compute results byprocessing input data based on weights associated with the trainedneural network. As used herein, the term ‘weight’ is an example of a‘parameter’ as used in various forms of neural network processing. Forexample, some neural network learning is directed to determiningparameters that are then usable for performing neural network inferencesusing the parameters.

A neural network processes data according to a dataflow graph comprisinglayers of neurons. Stimuli (e.g., input data) are received by an inputlayer of neurons and the computed results of the dataflow graph (e.g.,output data) are provided by an output layer of neurons. Example layersof neurons include input layers, output layers, rectified linear unitlayers, fully connected layers, recurrent layers, long short-term memorylayers, convolutional layers, kernel layers, dropout layers, and poolinglayers. A neural network is conditionally and/or selectively trained,subject to hardware acceleration. After being trained, a neural networkis conditionally and/or selectively used for inference, subject tohardware acceleration.

An example of a deep learning accelerator is one or more relativelyspecialized hardware elements operating in conjunction with one or moresoftware elements to train a neural network and/or perform inferencewith a neural network relatively more efficiently than using relativelyless specialized hardware elements. Some implementations of therelatively specialized hardware elements include one or more hardwarelogic circuitry elements such as transistors, resistors, inductors,capacitors, wire interconnects, combinatorial logic (e.g., NAND, NOR)gates, latches, register files, memory arrays, tags for memory arrays,content-addressable memories, flash, ROM, DRAM, SRAM,Serializer/Deserializer (SerDes), I/O drivers, and the like, such asimplemented via custom logic, synthesized logic, ASICs, and/or FPGAs.Some of the relatively less specialized hardware elements includeconventional CPUs and conventional GPUs.

An example of storage is one or more elements enabled to retain stateinformation, e.g., any one or more of: a flip-flop, a latch or an arrayof latches, a register or an array of registers, a register file, amemory, a memory array, a magnetic storage device, an optical storagedevice, SRAM, DRAM, flash, and ROM. In various embodiments storage isvolatile (e.g., SRAM or DRAM) and/or non-volatile (e.g., flash or ROM).

An example of an Integrated Circuit (IC) is a collection of circuitryimplemented on one or more portions of semiconductor material, such as asingle die or a plurality of dice. An example of 3D-stacking of dice isproviding mechanical connectivity and/or electrical connectivity betweenthe dice, e.g., in a dimension orthogonal to a major surface of thedice, to form a unit. The mechanical connectivity and/or the electricalconnectivity are variously implemented, e.g., via one or more of solderballs, microbumps, and through-silicon vias. An example of 2.5D stackingof dice is providing mechanical connectivity and/or electricalconnectivity between the dice via a common element (e.g., a siliconinterposer) to form a unit, wherein the mechanical connectivity and/orelectrical connectivity between each die and the common substrate is ina dimension orthogonal to a major surface of the die. The mechanicalconnectivity and/or the electrical connectivity are variouslyimplemented, e.g., via one or more of solder balls, microbumps, andthrough-silicon vias. An example of an Application-Specific IntegratedCircuit (ASIC) is an IC designed for a particular use.

An example of a package is an element enabled to mechanically retainand/or contain one or more electronic circuits and/or to electricallyinterconnect one or more electronic circuits. Example electroniccircuits are any one or more of one or more portions of semiconductormaterial, one or more dice, one or more interposers, and one or moresubstrates. Particular examples of packages include a BGA package andvariants thereof. Some ICs comprise a package. An example of a substrateis an element to mechanically retain and/or electrically interconnectone or more dice and/or one or more packages. A particular example of asubstrate is a PCB to, e.g., retain and interconnect packages. Anotherparticular example of a substrate is a silicon interposer to, e.g.,couple one or more 3D-stacked or 2.5-stacked dice. Another particularexample of a substrate is a package, e.g., retaining a plurality ofdice.

A SmartNIC is a network interface card, or network adapter that operatesdirectly on data packets independent of host kernel resources andrunning an operating system networking stack resulting in lesscontention for the host processing resources, less network latency, andincreases in network data packet throughput. The SmartNIC accomplishesthis by offloading network stack processing tasks from the system hostCPU, acting as a coprocessor of sorts.

In the present context, a SmartNIC is a NIC equipped with a fullyprogrammable hardware implementation, supporting an operating systemconfigured for network processing tasks. The hardware implementation maycomprise System-on-Chip (SoC), FPGAs, ASICs, CGRAs, or otherprogrammable processor circuits such as the ARM family. A SmartNIC maysupport sets of specialized hardware functionalities accelerates aspecific class of functions (e.g., Open vSwitch data-plane) or toperform generic packet and flow-filtering, packet inspection, flow tableprocessing, encryption, RDMA, VXLAN overlays and NVMe-oF functionality.

A SmartNIC includes a host kernel-bypass logic for sending and receivingpackets to/from nodes and additional hosts. The SmartNIC may accomplishthis by providing a set of physical addresses comprising a shared memoryfor inputs and outputs. In one aspect, the reprogrammable processor maydirectly access sets of SmartNIC FIFO buffers using a combination ofhead and tail pointers as described supra to push and pull data, thusbypassing the host kernel and reducing at least one hop. A host may alsointerface directly to the SmartNIC by writing to a physical addresswithout requiring drivers to control the network flow, furtherincreasing theoretical throughput.

In one aspect, the SmartNIC may provide a configuration interface tospecify the physical addresses of a plurality of I/O shared memorybuffers comprising FIFO queues and mapping tables for memory regionscontaining packet buffers. In an additional aspect, the SmartNIC maycouple nodes, reprogrammable processors (RPs) and hosts to retrievepacket buffers from shared memory buffers and to transmit packet buffersfrom host, node, or RP DRAM to the SmartNIC shared memory buffers over anetwork.

The network fabric is an interface to a plurality of nodes and hosts.The SmartNIC provides connectivity between either a host and the networkor between a node and the network. A node comprises a plurality ofreprogrammable processors (RPs) and bypasses the host when interfacingto the SmartNIC. A SmartNIC may connect to a first physical/linkconnection over the network, coupling the SmartNIC with a host, node, orRP. The SmartNIC connects to a second physical/link connection, couplingthe SmartNIC to the network. The physical/link connections to thenetwork fabric interface may each be of any type, for instance,Ethernet, Fibre Channel, InfiniBand, PCIe, etc. A physical/linkconnection may also be a wireless medium. A SmartNIC includes MediaAccess Controllers (MACs) to interface with the physical/linkconnections to route data packets to the RPs and hosts.

An example SmartNIC may use an FPGA to implement the communicationsprotocols, e.g., Transport Control Protocol (“TCP”), used to performinternet routing and may comprise PCIe high-speed network interfaces,shared physical memory and an FPGA. The FPGA may implement the SmartNICcontroller as the bridge between a host, node, RP, and the network atthe “physical layer” to integrate directly into the data path. TheSmartNIC may further implement the Open System Interconnection (“OSI”)model, which is a conceptual model that characterizes and standardizesthe internal functions of a communication system by partitioning it intoabstraction layers. A physical abstraction layer defines electrical andphysical specifications between a device and a transmission medium, suchas a copper or fiber optical cable. This includes the layout of pins,voltages, line impedance, cable specifications, signal timing, hubs,repeaters, network adapters, host bus adapters and more. The majorfunctions and services performed by the physical layer include: (1)establishment and termination of a connection to a communicationsmedium; (2) contention resolution; (3) flow control; and (4) modulationto convert digital data in user equipment to the corresponding signalstransmitted over a communications channel. These are the signalsoperating over the physical cabling (such as copper and optical fiber)or over a radio link.

The network flows can be Transmission Control Protocol/Internet Protocol(TCP/IP) flows, for example. The SmartNICs may exchange network packetswith the nodes or hosts via a network/fabric comprising media/physicallinks and can exchange network packets with their respective nodes orhosts via host-facing media/physical links to the host NICs. Networkflows used by applications to exchange data may pass through theSmartNIC as follows. A host-based application may have application-layerdata to convey, for instance, a remote call invocation. The host remotecall invocation may comprise a command or data for passing through anoperating system Application Programming Interface (API) (e.g., a streamor socket) as a write to a physical address on the SmartNIC where itenters the network stack, The API writes the command or data into thephysical address of the shared memory FIFO and placed in one or moretransport packets (e.g., TCP/IP packets). Next, encapsulation oftransport packets to network packets (e.g., TCP/IP packets with thehost's Internet Protocol (IP) address as the sender). and then loadedinto one or more payloads of physical layer frames (e.g., Ethernetframes). The frames then pass through to the first physical/linkconnection of the network fabric. On a second SmartNIC, the aboveprocess is reversed where the network packets require decapsulation anddata eventually arrives at a physical address for the host, node, or RP.

The applications execute on the reconfigurable processors in adistributed fashion by programming the individual compute and memorycomponents and may asynchronously receive, process, and send data andcontrol information. In the reconfigurable processors, computation mayexecute as deep, nested dataflow pipelines that exploit nestedparallelism and data locality efficiently. These dataflow pipelinescontain several stages of computation, where each stage reads data fromone or more input buffers with an irregular memory access pattern,performs computations on the data while using one or more internalbuffers to store and retrieve intermediate results, and produces outputsthat are written to one or more output buffers. The structure of thesepipelines depends on the control and dataflow graph representing theapplication. Pipelines may arbitrarily nest and loop within each other.

The applications/graphs/application graphs/user applications/dataflowgraphs/control flow graphs/data and control flow graphs/models/deeplearning applications/deep neural networks/programs/programimages/jobs/tasks comprise high-level programs. A high-level program issource code written in programming languages like C, C++, Java,JavaScript, Python, and Spatial, for example, using deep learningframeworks like PyTorch, TensorFlow, ONNX, Caffe, and Keras. Thehigh-level program can implement computing structures and algorithms ofmachine learning models like AlexNet, VGGNet, GoogLeNet, ResNet,ResNeXt, RCNN, YOLO, SqueezeNet, SegNet, GAN, BERT, ELMo, USE,Transformer, and Transformer-XL. In one example, the high-level programcan implement a convolutional neural network with several processinglayers, such that each processing layer can include one or more nestedloops. The high-level program can execute irregular memory operationsthat involve accessing inputs and weights and performing matrixmultiplications between the inputs and the weights. The high-levelprogram can include nested loops with high iteration count and loopbodies that load and multiply input values from a preceding processinglayer with weights of a succeeding processing layer to produce an outputfor the succeeding processing layer. The high-level program can haveloop-level parallelism of the outermost loop body, which can beexploited using coarse-grained pipelining. The high-level program canhave instruction-level parallelism of the innermost loop body, which canbe exploited using loop unrolling, SIMD vectorization, and pipelining.

Regarding loops in the high-level programs of the applications, loopsdirectly nested in a loop body are termed the child loops of the outerparent loop. A loop is called an innermost loop if it does not have anychildren, i.e., there are no nested loops within its body. A loop is anoutermost loop if it does not have a parent, i.e., it is not nestedwithin another loop's body. An imperfectly nested loop has a body with amix of non-looping statements (e.g., primitive arithmetic, logical, andrelational operations) and one or more child loops. Parallelism in theimperfectly nested loops can be exploited at any or all loop levels, andin the operations that comprise loop bodies. Parallelism can occur inmultiple forms such as fine-grained and coarse-grained pipelineparallelism, data parallelism, and task parallelism.

In some implementations, a Software Development Kit (SDK) (or dataflowgraph generator) generates dataflow graphs of the high-level programs ofthe applications. The SDK transforms the input behavioral description ofthe high-level programs into an intermediate representation such as thedataflow graphs. This may include code optimization steps like falsedata dependency elimination, dead-code elimination, and constantfolding. The dataflow graphs encode the data and control dependencies ofthe high-level programs.

The dataflow graphs comprise nodes and edges. The nodes can representcompute operations and memory allocations. The edges can representdataflow and control flow. In some implementations, each loop in thehigh-level programs can be represented as a controller in the dataflowgraphs. The dataflow graphs support branches, loops, function calls, andother variations of control dependencies. In some implementations, afterthe dataflow graphs are generated, additional analyses or optimizationsfocused on loop transformations can be performed, such as loopunrolling, loop pipelining, loop fission/fusion, and loop tiling.

The SDK also supports programming the reconfigurable processors in thepool of reconfigurable dataflow resources at multiple levels, forexample, from the high-level deep learning frameworks to C++ andassembly language. In some implementations, the SDK allows programmersto develop code that runs directly on the reconfigurable processors. Inother implementations, the SDK provides libraries that containpre-defined functions like linear algebra operations, element-wisetensor operations, non-linearities, and reductions required forcreating, executing, and profiling the dataflow graphs on thereconfigurable processors. The SDK communicates with the deep learningframeworks via Application Programming Interfaces (APIs).

The nodes in a dataflow graph represent operation units may configure tobe producers to produce tensors for execution of an application, and tobe consumers to consume the tensors for execution of the application.The producers and consumers asynchronously transmit data along dataconnections. A tensor includes one or more vectors.

A “compiler” transforms the dataflow graphs into a hardware-specificconfiguration, and specifies in an execution file generated by thecompiler 114. In one implementation, the compiler partitions thedataflow graphs into memory allocations and execution fragments, wherethese partitions are specified in the execution file. Executionfragments represent operations on data. An execution fragment cancomprise portions of a program representing an amount of work. Anexecution fragment can comprise computations encompassed by a set ofloops, a set of graph nodes, or some other unit of work that requiressynchronization. An execution fragment can comprise a fixed or variableamount of work, as needed by the program. Different ones of theexecution fragments can contain different amounts of computation.Execution fragments can represent parallel patterns or portions ofparallel patterns and are executable asynchronously.

In some implementations, the partitioning of the dataflow graphs intothe execution fragments includes treating calculations within at leastone innermost loop of a nested loop of the dataflow graphs as a separateexecution fragment. In other implementations, the partitioning of thedataflow graphs into the execution fragments includes treatingcalculations of an outer loop around the innermost loop of the dataflowgraphs as a separate execution fragment. In the case of imperfectlynested loops, operations within a loop body up to the beginning of anested loop within that loop body are grouped together as a separateexecution fragment.

Memory allocations represent the creation of logical memory spaces inon-chip and/or off-chip memories for data required to implement thedataflow graphs, and these memory allocations are specified in theexecution file. Memory allocations define the type and the number ofhardware resources (functional units, storage, or connectivitycomponents). Main memory (e.g., DRAM) is off-chip memory for providingmemory allocations. Scratchpad memory (e.g., SRAM) is on-chip memory forproviding memory allocations. Other memory types for which the memoryallocations can be made for various access patterns and layouts includeread-only Look-Up Tables (LUTs), fixed size queues (e.g., FIFOs), andregister files.

The compiler binds memory allocations to virtual memory units and bindsexecution fragments to virtual compute units, and these bindings arespecified in the execution file. In some implementations, the compilerpartitions execution fragments into memory fragments and computefragments, and these partitions are specified in the execution file. Amemory fragment comprises address calculations leading up to a memoryaccess. A compute fragment comprises all other operations in the parentexecution fragment. In one implementation, each execution fragment isbroken up into a plurality of memory fragments and exactly one computefragment. In one implementation, the compiler performs the partitioningusing reverse dataflow analysis such that inputs to an address used in amemory access recursively flag until the compiler reaches eitherconstant values or (bound) loop/pattern iterators. A single executionfragment can produce one or more memory fragments, depending on how manymemory accesses exist in the original loop body. In cases where the samememory addressing logic is shared across multiple memory accesses,address calculation may be duplicated to create multiple memoryfragments from the same execution fragment.

The memory fragments of the execution fragments are configured to indexinto data structures. At least one of the memory fragments indexes intoa data structure in the logical memory spaces of one of the memoryallocations. Each compute and memory fragment preserves informationabout all loops whose loop bodies directly contain the operations in thecorresponding execution fragment. In one implementation, thiscorresponds to replicating the calculation of the loop iterators of eachloop into each compute and memory fragment. This replication allows eachfragment to preserve the same iterative behavior as the originalprogram, while also allowing distributed calculation of loop iterators.

The compiler translates the applications developed with commonly usedopen-source packages such as Keras and PyTorch into reconfigurableprocessor specifications. The compiler generates the configuration fileswith configuration data (e.g., bit stream) for the placed positions andthe routed data and control networks. In one implementation, thisincludes assigning coordinates and communication resources of thephysical memory and compute units by placing and routing units onto thearray of the processor while maximizing bandwidth and minimizinglatency.

CLAUSES

A technology is described which uses buffers to efficiently stream databetween processors on a same processing node and on different processingnodes, which can be particularly applied to processors such as CentralProcessing Unit (CPUs), Graphics Processing Units (GPUs), FieldProgrammable Gate Arrays (FPGAs), Coarse-Grained ReconfigurableArchitectures (CGRAs), Application-Specific Integrated Circuits (ASICs),Application Specific Instruction-set Processor (ASIP), and DigitalSignal Processors (DSPs). The technology disclosed implements efficientdistributed computing by allowing accelerators (e.g., reconfigurableprocessors) attached to separate hosts to directly communicate with eachother via buffers.

The technology disclosed can be practiced as a system, method, orarticle of manufacture. One or more features of an implementation can becombined with the base implementation. Implementations that are notmutually exclusive are taught to be combinable. One or more features ofan implementation can be combined with other implementations. Thisdisclosure periodically reminds the user of these options. Omission fromsome implementations of recitations that repeat these options should notbe taken as limiting the combinations taught in the precedingsections—these recitations are hereby incorporated forward by referenceinto each of the following implementations.

One or more implementations and clauses of the technology disclosed orelements thereof can be implemented in the form of a computer product,including a non-transitory computer readable storage medium withcomputer usable program code for performing the method steps indicated.Furthermore, one or more implementations and clauses of the technologydisclosed or elements thereof can be implemented in the form of anapparatus including a memory and at least one processor that is coupledto the memory and operative to perform exemplary method steps. Yetfurther, in another aspect, one or more implementations and clauses ofthe technology disclosed or elements thereof can be implemented in theform of means for carrying out one or more of the method steps describedherein; the means can include (i) hardware module(s), (ii) softwaremodule(s) executing on one or more hardware processors, or (iii) acombination of hardware and software modules; any of (i)-(iii) implementthe specific techniques set forth herein, and the software modules arestored in a computer readable storage medium (or multiple such media).

The clauses described in this section can be combined as features. Inthe interest of conciseness, the combinations of features are notindividually enumerated and are not repeated with each base set offeatures. The reader will understand how features identified in theclauses described in this section can readily be combined with sets ofbase features identified as implementations in other sections of thisapplication. These clauses are not meant to be mutually exclusive,exhaustive, or restrictive; and the technology disclosed is not limitedto these clauses but rather encompasses all possible combinations,modifications, and variations within the scope of the claimed technologyand its equivalents.

Other implementations of the clauses described in this section caninclude a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the clausesdescribed in this section. Yet another implementation of the clausesdescribed in this section can include a system including memory and oneor more processors operable to execute instructions, stored in thememory, to perform any of the clauses described in this section.

We disclose the following clauses:

Clause Set 1

1. A data processing system, comprising:

a plurality of reconfigurable processors, reconfigurable processors inthe plurality of reconfigurable processors having reconfigurableprocessor memory;

a plurality of host processors, a host processor in the plurality ofhost processors operatively coupled to the reconfigurable processors andhaving host memory;

a plurality of buffers, buffers in the plurality of buffers includingreconfigurable processors-to-host processor buffers configured toreceive data from the reconfigurable processors and provide the data tothe host processor, and host processor-to-reconfigurable processorsbuffers configured to receive data from the host processor and providethe data to the reconfigurable processors;

runtime logic, running on the host processor, configured to load andexecute one or more configuration files for applications on thereconfigurable processors;

the reconfigurable processors configured to process the configurationfiles and data (e.g., weights, coefficients, vectors, tensors (imagedata, audio data, natural language processing (NLP data), control data(e.g., control tokens)) for the applications and generate outputs, andto send the outputs to the host processor using at least one of thereconfigurable processors-to-host processor buffers; and

debugging logic, running on the host processor, configured to detecterrors.

2. The data processing system of clause 1, wherein the debugging logicis further configured to report the errors to a debugging console on thehost processor based on comparison of the outputs to expected outputs.3. The data processing system of any of clauses 1-2, wherein thedebugging logic is further configured to report the errors to a debugoutput file on the host processor based on the comparison.4. The data processing system of any of clauses 1-3, wherein the buffersare accessible to the reconfigurable processors and the host processor.5. The data processing system of any of clauses 1-4, wherein the buffersare in the reconfigurable processor memory, and the reconfigurableprocessor memory is accessible to the host processor.6. The data processing system of any of clauses 1-5, wherein the buffersare in the host memory, and the host memory is accessible to thereconfigurable processors and the host processor.7. The data processing system of any of clauses 1-6, wherein the buffersare in a Network Interface Controller (NIC) that is accessible to thereconfigurable processors and the host processor.8. The data processing system of any of clauses 1-7, wherein the buffersare First-In, First-Out (FIFO) buffers.9. The data processing system of any of clauses 1-8, wherein thereconfigurable processors notify the host processor of error reportingusing one or more remote procedure calls.10. The data processing system of any of clauses 1-9, wherein thereconfigurable processors use at least one of the reconfigurableprocessors-to-host processor buffers to send one or more argument valuesto the host processor for execution of the remote procedure calls.11. The data processing system of any of clauses 1-10, wherein theruntime logic is further configured to execute one or more testconfiguration files for test applications on the reconfigurableprocessors, wherein the reconfigurable processors are further configuredto process the test configuration files data (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) for the testapplications and generate test outputs, and to send the test outputs tothe host processor using at least one of the reconfigurableprocessors-to-host processor buffers, and wherein testing logic, runningon the host processor, is configured to determine test statistics basedon the test outputs, and to report the test statistics to a test outputfile on the host processor.12. A data processing system, comprising:

a plurality of reconfigurable processors, reconfigurable processors inthe plurality of reconfigurable processors having reconfigurableprocessor memory;

a plurality of host processors, a host processor in the plurality ofhost processors operatively coupled to the reconfigurable processors andhaving host memory;

a plurality of buffers, buffers in the plurality of buffers includingreconfigurable processors-to-host processor buffers configured toreceive data from the reconfigurable processors and provide the data tothe host processor, and host processor-to-reconfigurable processorsbuffers configured to receive data from the host processor and providethe data to the reconfigurable processors;

runtime logic configured to load one or more configuration files forapplications on the reconfigurable processors for execution, theconfiguration files including a plurality of functions; and

the runtime logic configured to execute a first set of functions in theplurality of functions and/or data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on thereconfigurable processors, and a second set of functions in theplurality of functions and/or data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on the hostprocessor,

-   -   wherein functions in the second set of functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) are transmitted to the host        processor using one or more of the reconfigurable        processors-to-host processor buffers, and    -   wherein results of executing the functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the host processor are        transmitted to the reconfigurable processors using one or more        of the host processor-to-reconfigurable processors buffers.        13. The data processing system of any of clauses 1-12, wherein        the data on which the functions are executed is transmitted to        the host processor using the one or more of the reconfigurable        processors-to-host processor buffers.        14. The data processing system of any of clauses 1-13, further        comprising using respective ones of the reconfigurable        processors-to-host processor buffers to transmit respective        functions in the second set of functions and/or data therefor        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) to the host processor.        15. The data processing system of any of clauses 1-14, further        comprising using respective ones of the host        processor-to-reconfigurable processors buffers to transmit        results of executing the respective functions.        16. The data processing system of any of clauses 1-15, wherein        the buffers are accessible to the reconfigurable processors and        the host processor.        17. The data processing system of any of clauses 1-16, wherein        the buffers are in the reconfigurable processor memory, and the        reconfigurable processor memory is accessible to the host        processor.        18. The data processing system of any of clauses 1-17, wherein        the buffers are in the host memory, and the host memory is        accessible to the reconfigurable processors and the host        processor.        19. The data processing system of any of clauses 1-18, wherein        the buffers are in a Network Interface Controller (NIC) that is        accessible to the reconfigurable processors and the host        processor.        20. The data processing system of any of clauses 1-19, wherein        the buffers are First-In, First-Out (FIFO) buffers.        21. A data processing system, comprising:

a plurality of reconfigurable processors including a firstreconfigurable processor and additional reconfigurable processors;

a plurality of buffers, buffers in the plurality of buffers includingfirst reconfigurable processor-to-additional reconfigurable processorsbuffers configured to receive data from the first reconfigurableprocessor and provide the data to the additional reconfigurableprocessors, and additional reconfigurable processors-to-firstreconfigurable processor buffers configured to receive data from theadditional reconfigurable processors and provide the data to the firstreconfigurable processor;

runtime logic configured to load one or more configuration files forapplications on the first reconfigurable processor for execution, theconfiguration files including a plurality of functions; and

the runtime logic configured to execute a first set of functions in theplurality of functions and/or data therefor (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on the firstreconfigurable processor, and a second set of functions in the pluralityof functions and/or data therefor (e.g., weights, coefficients, vectors,tensors (image data, audio data, natural language processing (NLP data),control data (e.g., control tokens)) on the additional reconfigurableprocessors,

-   -   wherein functions in the second set of functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) are transmitted to the        additional reconfigurable processors using one or more of the        first reconfigurable processor-to-additional reconfigurable        processors buffers, and    -   wherein results of executing the functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the additional        reconfigurable processors are transmitted to the first        reconfigurable processor using one or more of the additional        reconfigurable processors-to-first reconfigurable processor        buffers.        22. The data processing system of any of clauses 1-21, wherein        the first reconfigurable processor and the additional        reconfigurable processors are operatively coupled to a same        processing node.        23. The data processing system of any of clauses 1-22, wherein        the first reconfigurable processor and the additional        reconfigurable processors are operatively coupled to different        processing nodes.        24. A data processing system, comprising:

a reconfigurable processor configured to execute one or moreconfiguration files using a series of data units;

a first plurality of buffers configured to receive data units in theseries of data units from a source memory, and to stream the data unitsto the reconfigurable processor for processing;

a second plurality of buffers configured to stream results of processingthe data units from the reconfigurable processor, and to send theresults to a destination memory for storage; and

runtime logic configured to cause buffers in the first plurality ofbuffers to receive a next data unit in the series of data units from thesource memory while the reconfigurable processor processes a currentdata unit in the series of data units, and to stream the next data unitto the reconfigurable processor for processing after buffers in thesecond plurality of buffers stream results of processing the currentdata unit from the reconfigurable processor.

25. The data processing system of any of clauses 1-24, wherein theruntime logic is further configured to cause the reconfigurableprocessor to process one or more previous data units while thereconfigurable processor is processing the current data unit.25. The data processing system of any of clauses 1-25, wherein theruntime logic is further configured to cause the buffers in the firstplurality of buffers to receive the next data unit from the sourcememory before the reconfigurable processor starts processing the currentdata unit.26. The data processing system of any of clauses 1-26, wherein theruntime logic is further configured to cause the buffers in the firstplurality of buffers to receive the next data unit from the sourcememory after the buffers in the second plurality of buffers stream theresults of processing the current data unit from the reconfigurableprocessor.28. A data processing system, comprising:

a plurality of reconfigurable processors, reconfigurable processors inthe plurality of reconfigurable processors having reconfigurableprocessor memory;

a plurality of host processors, a host processor in the plurality ofhost processors operatively coupled to the reconfigurable processors andhaving host memory;

a plurality of buffers in a shared memory accessible to thereconfigurable processors and the host processor; and

runtime logic configured to execute one or more configuration files thatdefine applications and process application data (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) for theapplications using the reconfigurable processors and the host processor,and

-   -   wherein execution of the configuration files and processing of        the application data includes receiving configuration data in        the configuration files and the application data from at least        one of the reconfigurable processors and providing the        configuration data and the application data to the host        processor, and receiving the configuration data and the        application data from the host processor and providing the        configuration data and the application data to the at least one        of the reconfigurable processors.        29. A data processing system, comprising:

a plurality of reconfigurable processors including a firstreconfigurable processor and additional reconfigurable processors;

a plurality of buffers in a shared memory accessible to the firstreconfigurable processor and the additional reconfigurable processors;and

runtime logic configured to execute one or more configuration files thatdefine applications and process application data (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) for theapplications using the first reconfigurable processor and the additionalreconfigurable processors, and

-   -   wherein execution of the configuration files and processing of        the application data includes receiving configuration data in        the configuration files and the application data from the first        reconfigurable processor and providing the configuration data        and the application data to at least one of the additional        reconfigurable processors, and receiving the configuration data        and the application data from the at least one of the additional        reconfigurable processors and providing the configuration data        and the application data to the first reconfigurable processor.

Clause Set 2

1. A data processing system, comprising:

a pool of reconfigurable dataflow resources including a plurality ofprocessing nodes, respective processing nodes in the plurality ofprocessing nodes operatively coupled to respective pluralities ofreconfigurable processors and respective pluralities of buffers; and

a runtime processor operatively coupled to the pool of reconfigurabledataflow resources, and configured to:

-   -   receive a plurality of configuration files for applications,        configuration files in the plurality of configuration files        specifying configurations of virtual dataflow resources required        to execute the configuration files, and the virtual dataflow        resources including a first virtual reconfigurable processor in        a first virtual processing node, a second virtual reconfigurable        processor in a second virtual processing node, and virtual        buffers that stream data between the first virtual        reconfigurable processor and the second virtual reconfigurable        processor;    -   allocate reconfigurable dataflow resources in the pool of        reconfigurable dataflow resources to the virtual dataflow        resources, the allocated reconfigurable dataflow resources        including        -   a first processing node in the respective processing nodes            allocated to the first virtual processing node,        -   a second processing node in the respective processing nodes            allocated to the second virtual processing node,        -   a first reconfigurable processor, operatively coupled to the            first processing node, allocated to the first virtual            reconfigurable processor, a second reconfigurable processor            operatively coupled to the second processing node allocated            to the second virtual reconfigurable processor, and        -   a first plurality of buffers, operatively coupled to the            first processing node, and a second plurality of buffers,            operatively coupled to the second processing node, allocated            to the virtual buffers; and    -   execute the configuration files and process data (e.g., weights,        coefficients, vectors, tensors (image data, audio data, natural        language processing (NLP data), control data (e.g., control        tokens)) for the applications using the allocated reconfigurable        dataflow resources.        2. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or clause 1 in Clause Set 2, wherein the first        plurality of buffers includes a first set of sender buffers        configured to receive data from the first reconfigurable        processor and provide the data to a second set of receiver        buffers in the second plurality of buffers, the second set of        receiver buffers configured to provide the data to the second        reconfigurable processor.        3. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-2 in Clause Set 2, wherein the        second plurality of buffers includes a second set of sender        buffers configured to receive data from the second        reconfigurable processor and provide the data to a first set of        receiver buffers in the first plurality of buffers, the first        set of receiver buffers configured to provide the data to the        first reconfigurable processor.        4. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-3 in Clause Set 2, wherein the        respective processing nodes are operatively coupled to        respective host processors.        5. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-4 in Clause Set 2, wherein the        first plurality of buffers operates in a memory of a first host        processor operatively coupled to the first processing node, and        the second plurality of buffers operates in a memory of a second        host processor operatively coupled to the second processing        node.        6. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-5 in Clause Set 2, wherein the        respective processing nodes are operatively coupled to        respective pluralities of Smart Network Interface Controllers        (SmartNICs).        7. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-6 in Clause Set 2, wherein the        first plurality of buffers operates in a memory of a first        SmartNIC operatively coupled to the first processing node.        8. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-7 in Clause Set 2, wherein the        runtime logic is further configured to configure the first        SmartNIC with a routing table that specifies the first        reconfigurable processor as a local reconfigurable processor,        and the second reconfigurable processor as a destination        reconfigurable processor.        9. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-8 in Clause Set 2, wherein the        second plurality of buffers operates in a memory of a second        SmartNIC operatively coupled to the second processing node.        10. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-9 in Clause Set 2, wherein the        runtime logic is further configured to configure the second        SmartNIC with a routing table that specifies the second        reconfigurable processor as a local reconfigurable processor,        and the first reconfigurable processor as a destination        reconfigurable processor.        11. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-10 in Clause Set 2, wherein at        least one of the applications is a dataflow graph with a set of        processing modules.        12. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-11 in Clause Set 2, wherein the        runtime logic is further configured to partition the set of        processing modules into a first subset of processing modules and        a second subset of processing modules.        13. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-12 in Clause Set 2, wherein the        runtime logic is further configured to execute configuration        files for the first subset of processing modules and therefor on        the first reconfigurable processor.        14. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-13 in Clause Set 2, wherein the        runtime logic is further configured to execute configuration        files for the second subset of processing modules and data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the second        reconfigurable processor.        15. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-14 in Clause Set 2, wherein the        runtime logic is further configured to use the first plurality        of buffers and the second plurality of buffers to stream data        between the first subset of processing modules and the second        subset of processing modules, wherein the data includes feature        maps and/or activations generated during a forward pass, and        loss gradients generated during a backward pass.        16. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-15 in Clause Set 2, wherein the        runtime logic is further configured to initialize a first        instance of the dataflow graph and a second instance of the        dataflow graph.        17. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-16 in Clause Set 2, wherein the        runtime logic is further configured to execute configuration        files for the first instance of the dataflow graph and data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the first reconfigurable        processor.        18. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-17 in Clause Set 2, wherein the        runtime logic is further configured to execute configuration        files for the second instance of the dataflow graph and data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the second        reconfigurable processor.        19. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-18 in Clause Set 2, wherein the        runtime logic is further configured to use the first plurality        of buffers and the second plurality of buffers to stream data        between the first instance of the dataflow graph and the second        instance of the dataflow graph, wherein the data includes        gradients generated during a backward pass.        20. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-19 in Clause Set 2, wherein the        first plurality of buffers operates in a memory of the first        reconfigurable processor, and the second plurality of buffers        operates in a memory of the second reconfigurable processor.        21. A data processing system, comprising:

a pool of reconfigurable dataflow resources including a plurality ofprocessing nodes, respective processing nodes in the plurality ofprocessing nodes operatively coupled to respective pluralities ofreconfigurable processors and respective pluralities of buffers; and

a runtime processor operatively coupled to the pool of reconfigurabledataflow resources, the runtime processor including runtime logicconfigured to:

-   -   receive a set of configuration files for an application;    -   load and execute a first subset of configuration files in the        set of configuration files and association application data        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) on a first reconfigurable processor        operatively coupled to a first processing node in the respective        processing nodes;    -   load and execute a second subset of configuration files in the        set of configuration files and associated application data        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) on a second reconfigurable processor        operatively coupled to a second processing node in the        respective processing nodes; and    -   use a first plurality of buffers operatively coupled to the        first processing node, and a second plurality of buffers        operatively coupled to the second processing node to stream data        between the first reconfigurable processor and the second        reconfigurable processor to load and execute the first subset of        configuration files and the second subset of configuration        files.        22. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-21 in Clause Set 2, wherein the        first plurality of buffers operates in a memory of a first host        processor operatively coupled to the first processing node, and        the second plurality of buffers operates in a memory of a second        host processor operatively coupled to the second processing        node.        23. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-22 in Clause Set 2, wherein the        first plurality of buffers operates in a memory of a first smart        Network Interface Controller (SmartNIC) operatively coupled to        the first processing node, and the second plurality of buffers        operates in a memory of a second SmartNIC operatively coupled to        the second processing node.        24. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-23 in Clause Set 2, wherein the        first plurality of buffers operates in a memory of the first        reconfigurable processor, and the second plurality of buffers        operates in a memory of the second reconfigurable processor.        25. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-24 in Clause Set 2, wherein a        network fabric operatively couples the first processing node and        the second processing node.        26. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-25 in Clause Set 2, wherein the        network fabric streams the data between the first plurality of        buffers and the second plurality of buffers.        27. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-26 in Clause Set 2, wherein the        runtime logic is further configured to:

load and execute a third subset of configuration files in the set ofconfiguration files and associated application data (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on a thirdreconfigurable processor operatively coupled to a third processing nodein the respective processing nodes;

load and execute a fourth subset of configuration files in the set ofconfiguration files and associated application data (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) on a fourthreconfigurable processor operatively coupled to a fourth processing nodein the respective processing nodes; and

use a third plurality of buffers operatively coupled to the thirdprocessing node, and a fourth plurality of buffers operatively coupledto the fourth processing node to stream data between the thirdreconfigurable processor and the fourth reconfigurable processor to loadand execute the third subset of configuration files and the fourthsubset of configuration files.

28. A data processing system, comprising:

a processing node operatively coupled to reconfigurable processors thathave different levels of configurable granularity; and

a runtime processor operatively coupled to the processing node, theruntime processor including runtime logic configured to:

-   -   receive a set of configuration files for an application;    -   load and execute a first subset of configuration files in the        set of configuration files and associated application data        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) on a first reconfigurable processor in        the reconfigurable processors, the first reconfigurable        processor having a first level of configurable granularity; and    -   load and execute a second subset of configuration files in the        set of configuration files and associated application data        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) on a second reconfigurable processor in        the reconfigurable processors, the second reconfigurable        processor having a second level of configurable granularity that        is different from the first level of configurable granularity.        29. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-28 in Clause Set 2, wherein the        first level of configurable granularity is a bit-level        configurable granularity, and the first reconfigurable processor        is a Field-Programmable Gate Array (FPGA).        30. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-29 in Clause Set 2, wherein the        second level of configurable granularity is a word-level        configurable granularity, and the second reconfigurable        processor is a Coarse-Grained Reconfigurable Architecture        (CGRA).        31. A data processing system, comprising:

a processing node operatively coupled to reconfigurable processors thathave different levels of configurable granularity; and

a runtime processor operatively coupled to the processing node, theruntime processor including runtime logic configured to:

-   -   receive a set of configuration files for an application;    -   load and execute a first subset of configuration files in the        set of configuration files and associated application data        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) on a first reconfigurable processor in        the reconfigurable processors, the first reconfigurable        processor having a first configuration; and    -   load and execute a second subset of configuration files in the        set of configuration files and associated application data        (e.g., weights, coefficients, vectors, tensors (image data,        audio data, natural language processing (NLP data), control data        (e.g., control tokens)) on a second reconfigurable processor in        the reconfigurable processors, the second reconfigurable        processor having a second configuration that is different from        the configuration.        32. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-31 in Clause Set 2, wherein the        first configuration is a bit-level configurable granularity, and        the first reconfigurable processor is a Field-Programmable Gate        Array (FPGA).        33. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-32 in Clause Set 2, wherein the        second configuration is a word-level configurable granularity,        and the second reconfigurable processor is a Coarse-Grained        Reconfigurable Architecture (CGRA).        34. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-33 in Clause Set 2, wherein the        first configuration is a gate-level reconfigurability, and the        first reconfigurable processor is the FPGA.        35. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-34 in Clause Set 2, wherein the        second configuration is a register transfer-level        reconfigurability, and the second reconfigurable processor is        the CGRA.        36. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-35 in Clause Set 2, wherein the        first configuration uses bit-wise Look-Up Tables (LUTs) and        switches, and the first reconfigurable processor is the FPGA.        37. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-36 in Clause Set 2, wherein the        second configuration uses word-wide Issue Slots (ISs)/Arithmetic        Logic Units (ALUs)/Functional Units (FUs)/Processing Elements        (PEs), Register Files (RFs), and interconnections, and the        second reconfigurable processor is the CGRA.        38. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-37 in Clause Set 2, wherein a        number of the ISs used by the second reconfigurable processor is        fewer than a number of the LUTs used by the first reconfigurable        processor.        39. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-38 in Clause Set 2, wherein a        number of bits required to configure the second reconfigurable        processor is orders of magnitude smaller than a number of bits        required to configure the first reconfigurable processor.

Clause Set 3

1. A data processing system, comprising:

a plurality of processing nodes, processing nodes in the plurality ofprocessing nodes including a first processing node and a secondprocessing node, the first processing node operatively coupled to thesecond processing node, the first processing node having a first hostprocessor, a first plurality of reconfigurable processors operativelycoupled to the first host processor, and a first plurality of SmartNetwork Interface Controllers (SmartNICs) operatively coupled to thefirst plurality of reconfigurable processors, and the second processingnode having a second host processor, a second plurality ofreconfigurable processors operatively coupled to the second hostprocessor, and a second plurality of SmartNICs operatively coupled tothe second plurality of reconfigurable processors;

a first plurality of buffers in a memory of a first SmartNIC in thefirst plurality of SmartNICs, the first SmartNIC operatively coupled toa first reconfigurable processor in the first plurality ofreconfigurable processors;

a second plurality of buffers in a memory of a second SmartNIC in thesecond plurality of SmartNICs, the second SmartNIC operatively coupledto a second reconfigurable processor in the second plurality ofreconfigurable processors;

the first plurality of buffers including a first set of sender buffersconfigured to receive data from the first reconfigurable processor andprovide the data to a second set of receiver buffers in the secondplurality of buffers, the second set of receiver buffers configured toprovide the data to the second reconfigurable processor;

the second plurality of buffers including a second set of sender buffersconfigured to receive data from the second reconfigurable processor andprovide the data to a first set of receiver buffers in the firstplurality of buffers, the first set of receiver buffers configured toprovide the data to the first reconfigurable processor; and

runtime logic configured to execute configuration files that defineapplications and process application data (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) for the applicationsusing the first reconfigurable processor and the second reconfigurableprocessor, the execution including streaming configuration data (e.g.,bit stream) in the configuration files and the application data betweenthe first reconfigurable processor and the second reconfigurableprocessor using one or more buffers in the first plurality of buffersand one or more buffers in the second plurality of buffers, thereby thestreaming bypassing the first host processor and the second hostprocessor.

2. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or clause 1 in Clause Set3, wherein the execution includes streaming input data for theapplications from the first reconfigurable processor to the secondreconfigurable processor.3. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-2 inClause Set 3, wherein one or more sender buffers in the first set ofsender buffers are configured to receive the input data from the firstreconfigurable processor and provide the input data to one or morereceiver buffers in the second set of receiver buffers, wherein thereceiver buffers in the second set of receiver buffers are configured toprovide the input data to the second reconfigurable processor.4. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-3 inClause Set 3, wherein the execution includes streaming output data forthe applications from the second reconfigurable processor to the firstreconfigurable processor, wherein the output data is generated as aresult of processing the input data.5. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-4 inClause Set 3, wherein one or more sender buffers in the second set ofsender buffers are configured to receive the output data from the secondreconfigurable processor and provide the output data to one or morereceiver buffers in the first set of receiver buffers, wherein thereceiver buffers in the first set of receiver buffers are configured toprovide the output data to the first reconfigurable processor.6. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-5 inClause Set 3, wherein the first reconfigurable processor is configuredto push the input data to the first SmartNIC, wherein the first SmartNICis configured to write the input data into the sender buffers in thefirst set of sender buffers, and wherein the first SmartNIC isconfigured to update tail pointers of the sender buffers in the firstset of sender buffers in response to the writing of the input data.7. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-6 inClause Set 3, wherein the first SmartNIC is configured to send the inputdata to the second SmartNIC in response to the updated tail pointers,wherein the second SmartNIC is configured to write the input data intothe receiver buffers in the second set of receiver buffers, and whereinthe second SmartNIC is configured to update tail pointers of thereceiver buffers in the second set of receiver buffers in response tothe writing of the input data.8. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-7 inClause Set 3, wherein the second reconfigurable processor is configuredto pull the input data from the second SmartNIC by reading the inputdata from the receiver buffers in the second set of receiver buffers inresponse to the updated tail pointers.9. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-8 inClause Set 3, wherein the second reconfigurable processor is configuredto push the output data to the second SmartNIC, wherein the secondSmartNIC is configured to write the output data into the sender buffersin the second set of sender buffers, and wherein the second SmartNIC isconfigured to update tail pointers of the sender buffers in the secondset of sender buffers in response to the writing of the output data.10. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-9 inClause Set 3, wherein the second SmartNIC is configured to send theoutput data to the first SmartNIC in response to the updated tailpointers, wherein the first SmartNIC is configured to write the outputdata into the receiver buffers in the first set of receiver buffers, andwherein the first SmartNIC is configured to update tail pointers of thereceiver buffers in the first set of receiver buffers in response to thewriting of the output data.11. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-38 in Clause Set 2 and/or any of clauses 1-10 inClause Set 3, wherein the first reconfigurable processor is configuredto pull the output data from the first SmartNIC by reading the outputdata from the receiver buffers in the first set of receiver buffers inresponse to the updated tail pointers.12. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-11 inClause Set 3, wherein a network fabric operatively couples the firstprocessing node and the second processing node, and the network fabricis configured to implement the streaming of the input data from thesender buffers in the first set of sender buffers to the receiverbuffers in the second set of receiver buffers, and to implement thestreaming of the output data from the sender buffers in the second setof sender buffers to the receiver buffers in the first set of receiverbuffers.13. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-12 inClause Set 3, wherein the first reconfigurable processor notifies thesecond reconfigurable processor of remote invocations using one or moreremote procedure calls.14. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-13 inClause Set 3, wherein the first reconfigurable processor uses the senderbuffers in the first set of sender buffers and the receiver buffers inthe second set of receiver buffers to send one or more argument valuesto the second reconfigurable processor for execution of the remoteprocedure calls.15. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-14 inClause Set 3, wherein the second reconfigurable processor notifies thefirst reconfigurable processor of remote invocations using one or moreremote procedure calls.16. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-15 inClause Set 3, wherein the second reconfigurable processor uses thesender buffers in the second set of sender buffers and the receiverbuffers in the first set of receiver buffers to send one or moreargument values to the first reconfigurable processor for execution ofthe remote procedure calls.17. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-16 inClause Set 3, wherein respective SmartNICs in the first plurality ofSmartNICs are operatively coupled to respective reconfigurableprocessors in the first plurality of reconfigurable processors byrespective buses, and respective SmartNICs in the second plurality ofSmartNICs are operatively coupled to respective reconfigurableprocessors in the second plurality of reconfigurable processors byrespective buses.18. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-17 inClause Set 3, wherein the configuration files include a plurality offunctions.19. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-18 inClause Set 3, further comprising:

the runtime logic configured to execute a first set of functions in theplurality of functions on the first reconfigurable processor, and asecond set of functions in the plurality of functions on the secondreconfigurable processor,

-   -   wherein functions in the second set of functions and/or data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) are transmitted to the        second reconfigurable processor using the sender buffers in the        first set of sender buffers and the receiver buffers in the        second set of receiver buffers, and    -   wherein results of executing the functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the second        reconfigurable processor are transmitted to the first        reconfigurable processor using the sender buffers in the second        set of sender buffers and the receiver buffers in the first set        of receiver buffers.        20. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-39 in Clause Set 2 and/or any of        clauses 1-19 in Clause Set 3, wherein the buffers in the first        plurality of buffers and the buffers in the second plurality of        buffers are First-In, First-Out (FIFO) buffers.        21. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-39 in Clause Set 2 and/or any of        clauses 1-20 in Clause Set 3, wherein the runtime logic runs in        at least one of the first host processor and the second host        processor.        22. A data processing system, comprising:

a plurality of processing nodes, processing nodes in the plurality ofprocessing nodes including a first processing node and a secondprocessing node, the first processing node operatively coupled to thesecond processing node, the first processing node having a first hostprocessor, a first plurality of reconfigurable processors operativelycoupled to the first host processor, and a first plurality of SmartNetwork Interface Controllers (SmartNICs) operatively coupled to thefirst plurality of reconfigurable processors, and the second processingnode having a second host processor, a second plurality ofreconfigurable processors operatively coupled to the second hostprocessor, and a second plurality of SmartNICs operatively coupled tothe second plurality of reconfigurable processors;

a first plurality of buffers in a memory of a first SmartNIC in thefirst plurality of SmartNICs, the first SmartNIC operatively coupled toa first reconfigurable processor in the first plurality ofreconfigurable processors;

a second plurality of buffers in a memory of a second SmartNIC in thesecond plurality of SmartNICs, the second SmartNIC operatively coupledto the second host processor;

the first plurality of buffers including a first set of sender buffersconfigured to receive data from the first reconfigurable processor andprovide the data to a second set of receiver buffers in the secondplurality of buffers, the second set of receiver buffers configured toprovide the data to the second host processor;

the second plurality of buffers including a second set of sender buffersconfigured to receive data from the second host processor and providethe data to a first set of receiver buffers in the first plurality ofbuffers, the first set of receiver buffers configured to provide thedata to the first reconfigurable processor; and

runtime logic configured to execute configuration files that defineapplications and process application data (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) for the applicationsusing the first reconfigurable processor and the second host processor,the execution including streaming configuration data (e.g., bit stream)in the configuration files and the application data between the firstreconfigurable processor and the second host processor using one or morebuffers in the first plurality of buffers and one or more buffers in thesecond plurality of buffers, thereby the streaming bypassing the firsthost processor.

23. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-22 inClause Set 3, wherein the first reconfigurable processor notifies thesecond host processor of remote invocations using one or more remoteprocedure calls.24. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-23 inClause Set 3, wherein the first reconfigurable processor uses one ormore sender buffers in the first set of sender buffers and one or morereceiver buffers in the second set of receiver buffers to send one ormore argument values to the second host processor for execution of theremote procedure calls.25. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-24 inClause Set 3, wherein the second host processor notifies the firstreconfigurable processor of remote invocations using one or more remoteprocedure calls.26. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-25 inClause Set 3, wherein the second host processor uses one or more senderbuffers in the second set of sender buffers and one or more receiverbuffers in the first set of receiver buffers to send one or moreargument values to the first reconfigurable processor for execution ofthe remote procedure calls.27. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-26 inClause Set 3, further comprising debugging logic configured to detecterrors, and report the errors to a debugging console on the second hostprocessor using the sender buffers in the first set of sender buffersand the receiver buffers in the second set of receiver buffers.28. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-27 inClause Set 3, further comprising:

the runtime logic configured to execute test configuration files thatdefine test applications and process application data (e.g., weights,coefficients, vectors, tensors (image data, audio data, natural languageprocessing (NLP data), control data (e.g., control tokens)) for the testapplications on the first reconfigurable processor; and

testing logic configured to generate results of execution of the testconfiguration files and the application data, and report the results toan output file on the second host processor using the sender buffers inthe first set of sender buffers and the receiver buffers in the secondset of receiver buffers.

29. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-28 inClause Set 3, wherein the configuration files include a plurality offunctions.30. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-29 inClause Set 3, further comprising:

the runtime logic configured to execute a first set of functions in theplurality of functions and data therefor (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) on the firstreconfigurable processor, and a second set of functions in the pluralityof functions and data therefor (e.g., weights, coefficients, vectors,tensors (image data, audio data, natural language processing (NLP data),control data (e.g., control tokens)) on the second host processor,

-   -   wherein functions in the second set of functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) are transmitted to the        second host processor using the sender buffers in the first set        of sender buffers and the receiver buffers in the second set of        receiver buffers, and    -   wherein results of executing the functions and/or the data        therefor (e.g., weights, coefficients, vectors, tensors (image        data, audio data, natural language processing (NLP data),        control data (e.g., control tokens)) on the second host        processor are transmitted to the first reconfigurable processor        using the sender buffers in the second set of sender buffers and        the receiver buffers in the first set of receiver buffers.        31. A data processing system, comprising:

a plurality of reconfigurable processors including a firstreconfigurable processor and a second reconfigurable processor;

a first Smart Network Interface Controller (SmartNIC) operativelycoupled to the first reconfigurable processor, the first SmartNIC havinga first plurality of buffers;

a second SmartNIC operatively coupled to the second reconfigurableprocessor, the second SmartNIC having a second plurality of buffers; and

runtime logic configured to execute configuration files that defineapplications and process application data (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) for the applicationsusing the first reconfigurable processor and the second reconfigurableprocessor, the execution including streaming configuration data (e.g.,bit stream) in the configuration files and the application data betweenthe first reconfigurable processor and the second reconfigurableprocessor using one or more buffers in the first plurality of buffersand one or more buffers in the second plurality of buffers.

32. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-31 inClause Set 3, wherein the first reconfigurable processor is on a firstprocessing node and operatively coupled to a first host processor,wherein the second reconfigurable processor is on a second processingnode and operatively coupled to a second host processor, and wherein thefirst processing node and the second processing node are operativelycoupled by a network fabric.33. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-32 inClause Set 3, wherein the first reconfigurable processor and the secondreconfigurable processor are on a same processing node and operativelycoupled to a same host processor.34. A data processing system, comprising:

a first reconfigurable processor operatively coupled to a first hostprocessor running on a first processing node;

a second reconfigurable processor operatively coupled to a second hostprocessor on a second processing node;

a first Smart Network Interface Controller (SmartNIC) operativelycoupled to the first reconfigurable processor, the first SmartNIC havinga first plurality of buffers;

a second SmartNIC operatively coupled to the second host processor, thesecond SmartNIC having a second plurality of buffers; and

runtime logic configured to execute configuration files that defineapplications and process application data (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) for the applicationsusing the first reconfigurable processor and the second host processor,the execution including streaming configuration data (e.g., bit stream)in the configuration files and the application data between the firstreconfigurable processor and the second host processor using one or morebuffers in the first plurality of buffers and one or more buffers in thesecond plurality of buffers.

35. A data processing system, comprising:

a first reconfigurable processor operatively coupled to a first hostprocessor running on a first processing node;

a second reconfigurable processor operatively coupled to a second hostprocessor on a second processing node;

a first Network Interface Controller (NIC) operatively coupled to thefirst processing node, the first NIC having a first plurality ofbuffers;

a second NIC operatively coupled to the second processing node, thesecond NIC having a second plurality of buffers; and

runtime logic configured to execute configuration files that defineapplications and application data (e.g., weights, coefficients, vectors,tensors (image data, audio data, natural language processing (NLP data),control data (e.g., control tokens)) for the applications using thefirst reconfigurable processor and the second reconfigurable processor,the execution including:

-   -   the first reconfigurable processor configured to push input data        for the applications to the one or more buffers in the first        plurality of buffers;    -   the first host processor configured to cause the first NIC to        stream the input data to one or more buffers in the second        plurality of buffers from the first plurality of buffers; and    -   the second host processor configured to cause the second NIC to        stream the input data to the second reconfigurable processor        from the buffers in the second plurality of buffers.        36. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-39 in Clause Set 2 and/or any of        clauses 1-35 in Clause Set 3, wherein the second host processor        uses one or more Remote Direct Memory Access (RDMA) commands to        update tail pointers of the buffers in the second plurality of        buffers after the input data is streamed to the buffers in the        second plurality of buffers.        37. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-39 in Clause Set 2 and/or any of        clauses 1-36 in Clause Set 3, wherein the second reconfigurable        processor is configured to pull the input data from the buffers        in the second plurality of buffers in response to the updated        tail pointers.        38. The data processing system of any of clauses 1-29 in Clause        Set 1 and/or any of clauses 1-39 in Clause Set 2 and/or any of        clauses 1-37 in Clause Set 3, the execution further including:

the second reconfigurable processor to push output data for theapplications to the buffers in the second plurality of buffers, whereinthe output data is generated as a result of processing the input data;

the second host processor configured to cause the second NIC to streamthe output data to the buffers in the first plurality of buffers fromthe second plurality of buffers; and

the first host processor configured to cause the first NIC to stream theoutput data to the first reconfigurable processor from the buffers inthe first plurality of buffers.

39. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-38 inClause Set 3, wherein the first host processor uses one or more RDMAcommands to update tail pointers of the buffers in the first pluralityof buffers after the output data is streamed to the buffers in the firstplurality of buffers.40. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-39 inClause Set 3, wherein the first reconfigurable processor is configuredto pull the output data from the buffers in the first plurality ofbuffers in response to the updated tail pointers.41. A data processing system, comprising:

a first reconfigurable processor having a first Network InterfaceController (NIC), and the first NIC having a first plurality of buffers;

a second reconfigurable processor having a second NIC, and the secondNIC having a second plurality of buffers; and

runtime logic configured to execute configuration files that defineapplications and process application data (e.g., weights, coefficients,vectors, tensors (image data, audio data, natural language processing(NLP data), control data (e.g., control tokens)) for the applicationsusing the first reconfigurable processor and the second reconfigurableprocessor, the execution including streaming configuration data (e.g.,bit stream) in the configuration files and the application data betweenthe first reconfigurable processor and the second reconfigurableprocessor using the first plurality of buffers of the first NIC and thesecond plurality of buffers of the second NIC.

42. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-41 inClause Set 3, wherein the first NIC is a first SmartNIC, and the secondNIC is a second SmartNIC, wherein the first and second reconfigurableprocessors are on a same processing node, and wherein the first andsecond reconfigurable processors are on different processing nodes.43. A data processing system, comprising:

a first reconfigurable processor operatively coupled to a first hostprocessor running on a first processing node, the first processing nodeoperatively coupled to a first Network Interface Controller (NIC);

a second reconfigurable processor operatively coupled to a second hostprocessor running on a second processing node, the second processingnode operatively coupled to a second NIC;

an address generator of the first reconfigurable processor configured tostream configuration data (e.g., bit stream) and application data (e.g.,weights, coefficients, vectors, tensors (image data, audio data, naturallanguage processing (NLP data), control data (e.g., control tokens)) forexecution of configuration files that define applications from the firstreconfigurable processor to the second reconfigurable processor usingmemory addresses that map to a first plurality of buffers; and

an address generator of the second reconfigurable processor configuredto stream the configuration data (e.g., bit stream) and the applicationdata from the second reconfigurable processor to the firstreconfigurable processor using memory addresses that map to a secondplurality of buffers.

44. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-43 inClause Set 3, wherein the first plurality of buffers operates in amemory of the first reconfigurable processor.45. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-44 inClause Set 3, wherein the first plurality of buffers operates in amemory of the first host processor.46. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-45 inClause Set 3, wherein the first plurality of buffers operates in amemory of the first NIC.47. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-46 inClause Set 3, wherein the second plurality of buffers operates in amemory of the second reconfigurable processor.48. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-47 inClause Set 3, wherein the second plurality of buffers operates in amemory of the second host processor.49. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-48 inClause Set 3, wherein the second plurality of buffers operates in amemory of the second NIC.50. The data processing system of any of clauses 1-29 in Clause Set 1and/or any of clauses 1-39 in Clause Set 2 and/or any of clauses 1-49 inClause Set 3, wherein the first NIC is a first SmartNIC, and the secondNIC is a second SmartNIC.

Clause Set 4

1. A computer-implemented method, including:

receiving a plurality of configuration files that define applications,configuration files in the plurality of configuration files specifyingconfigurations of virtual dataflow resources required to execute theconfiguration files, and the virtual dataflow resources including afirst virtual reconfigurable processor in a first virtual processingnode, a second virtual reconfigurable processor in a second virtualprocessing node, and virtual buffers that stream data between the firstvirtual reconfigurable processor and the second virtual reconfigurableprocessor;

allocating reconfigurable dataflow resources in a pool of reconfigurabledataflow resources to the virtual dataflow resources, the pool ofreconfigurable dataflow resources including a plurality of processingnodes, respective processing nodes in the plurality of processing nodesoperatively coupled to respective pluralities of reconfigurableprocessors and respective pluralities of buffers, the allocatedreconfigurable dataflow resources including

-   -   a first processing node in the respective processing nodes        allocated to the first virtual processing node,    -   a second processing node in the respective processing nodes        allocated to the second virtual processing node,    -   a first reconfigurable processor, operatively coupled to the        first processing node, allocated to the first virtual        reconfigurable processor,    -   a second reconfigurable processor operatively coupled to the        second processing node allocated to the second virtual        reconfigurable processor, and    -   a first plurality of buffers, operatively coupled to the first        processing node, and a second plurality of buffers, operatively        coupled to the second processing node, allocated to the virtual        buffers; and

executing the configuration files and processing application data forthe applications using the allocated reconfigurable dataflow resources.

2. A computer-implemented method, including:

-   -   receiving a set of configuration files for an application;    -   loading and executing a first subset of configuration files in        the set of configuration files and associated application data        on a first reconfigurable processor operatively coupled to a        first processing node in respective processing nodes;    -   loading and executing a second subset of configuration files in        the set of configuration files and associated application data        on a second reconfigurable processor operatively coupled to a        second processing node in the respective processing nodes; and    -   using a first plurality of buffers operatively coupled to the        first processing node, and a second plurality of buffers        operatively coupled to the second processing node to stream data        between the first reconfigurable processor and the second        reconfigurable processor to load and execute the first subset of        configuration files and the second subset of configuration        files.        3. A computer-implemented method, including:    -   receiving a set of configuration files for an application and        associated application data;    -   loading and executing a first subset of configuration files in        the set of configuration files and associated application data        on a first reconfigurable processor having a first level of        configurable granularity; and    -   loading and executing a second subset of configuration files in        the set of configuration files and associated application data        on a second reconfigurable processor having a second level of        configurable granularity that is different from the first level        of configurable granularity.        4. A computer-implemented method, including:    -   receiving a set of configuration files for an application and        associated application data;    -   loading and executing a first subset of configuration files in        the set of configuration files and associated application data        on a first reconfigurable processor having a first        configuration; and    -   loading and executing a second subset of configuration files in        the set of configuration files and associated application data        on a second reconfigurable processor having a second        configuration that is different from the first configuration.        5. A computer-implemented method, including:    -   executing configuration files that define applications and        processing application data for the applications using a first        reconfigurable processor and a second reconfigurable processor,        the execution including streaming configuration data (e.g., bit        stream) in the configuration files and the application data        between the first reconfigurable processor and the second        reconfigurable processor using one or more buffers in a first        plurality of buffers and one or more buffers in a second        plurality of buffers, thereby the streaming bypassing a first        host processor and a second host processor.        6. A computer-implemented method, including:    -   executing configuration files that define applications and        processing application data for the applications using a first        reconfigurable processor and a second host processor, the        execution including streaming configuration data (e.g., bit        stream) in the configuration files and the application data        between the first reconfigurable processor and the second host        processor using one or more buffers in a first plurality of        buffers and one or more buffers in a second plurality of        buffers, thereby the streaming bypassing a first host processor.        7. A data processing system, comprising:

a pool of reconfigurable dataflow resources including a plurality ofprocessing nodes, respective processing nodes in the plurality ofprocessing nodes operatively coupled to respective pluralities ofreconfigurable processors and respective pluralities of buffers; and

a runtime processor, running on one or more reconfigurable processors inthe respective pluralities of reconfigurable processors, and configuredto:

-   -   receive a plurality of configuration files for applications,        configuration files in the plurality of configuration files        specifying configurations of virtual dataflow resources required        to execute the configuration files, and the virtual dataflow        resources including a first virtual reconfigurable processor in        a first virtual processing node, a second virtual reconfigurable        processor in a second virtual processing node, and virtual        buffers that stream data between the first virtual        reconfigurable processor and the second virtual reconfigurable        processor;    -   allocate reconfigurable dataflow resources in the pool of        reconfigurable dataflow resources to the virtual dataflow        resources, the allocated reconfigurable dataflow resources        including        -   a first processing node in the respective processing nodes            allocated to the first virtual processing node,        -   a second processing node in the respective processing nodes            allocated to the second virtual processing node,        -   a first reconfigurable processor, operatively coupled to the            first processing node, allocated to the first virtual            reconfigurable processor,        -   a second reconfigurable processor operatively coupled to the            second processing node allocated to the second virtual            reconfigurable processor, and        -   a first plurality of buffers, operatively coupled to the            first processing node, and a second plurality of buffers,            operatively coupled to the second processing node, allocated            to the virtual buffers; and    -   execute the configuration files and process application data for        the applications using the allocated reconfigurable dataflow        resources.        8. A data processing system, comprising:

a pool of reconfigurable dataflow resources including a plurality ofprocessing nodes, respective processing nodes in the plurality ofprocessing nodes operatively coupled to respective pluralities ofreconfigurable processors and respective pluralities of buffers; and aruntime processor, running on one or more reconfigurable processors inthe respective pluralities of reconfigurable processors, and configuredto:

-   -   receive a set of configuration files for an application;    -   load and execute a first subset of configuration files in the        set of configuration files and associated application data on a        first reconfigurable processor operatively coupled to a first        processing node in the respective processing nodes;    -   load and execute a second subset of configuration files in the        set of configuration files and associated application data on a        second reconfigurable processor operatively coupled to a second        processing node in the respective processing nodes; and    -   use a first plurality of buffers operatively coupled to the        first processing node, and a second plurality of buffers        operatively coupled to the second processing node to stream data        between the first reconfigurable processor and the second        reconfigurable processor to load and execute the first subset of        configuration files and the second subset of configuration        files.        9. A data processing system, comprising:

a processing node operatively coupled to reconfigurable processors thathave different levels of configurable granularity; and

a runtime processor, running on one or more of the reconfigurableprocessors, the runtime processor including runtime logic configured to:

-   -   receive a set of configuration files for an application and        associated application data;    -   load and execute a first subset of configuration files in the        set of configuration files and associated application data on a        first reconfigurable processor in the reconfigurable processors,        the first reconfigurable processor having a first level of        configurable granularity; and    -   load and execute a second subset of configuration files in the        set of configuration files and associated application data on a        second reconfigurable processor in the reconfigurable        processors, the second reconfigurable processor having a second        level of configurable granularity that is different from the        first level of configurable granularity.        10. A data processing system, comprising:

a processing node operatively coupled to reconfigurable processors thathave different levels of configurable granularity; and

a runtime processor, running on one or more of the reconfigurableprocessors, the runtime processor including runtime logic configured to:

-   -   receive a set of configuration files for an application and        associated application data;    -   load and execute a first subset of configuration files in the        set of configuration files and associated application data on a        first reconfigurable processor in the reconfigurable processors,        the first reconfigurable processor having a first configuration;        and    -   load and execute a second subset of configuration files in the        set of configuration files and associated application data on a        second reconfigurable processor in the reconfigurable        processors, the second reconfigurable processor having a second        configuration that is different from the configuration.        11. A data processing system, comprising:

a plurality of processing nodes, processing nodes in the plurality ofprocessing nodes including a first processing node and a secondprocessing node, the first processing node operatively coupled to thesecond processing node, the first processing node having a firstplurality of reconfigurable processors operatively coupled and a firstplurality of Smart Network Interface Controllers (SmartNICs), and thesecond processing node having a second plurality of reconfigurableprocessors and a second plurality of SmartNICs;

a first plurality of buffers in a memory of a first SmartNIC in thefirst plurality of SmartNICs, the first SmartNIC operatively coupled toa first reconfigurable processor in the first plurality ofreconfigurable processors;

a second plurality of buffers in a memory of a second SmartNIC in thesecond plurality of SmartNICs, the second SmartNIC operatively coupledto a second reconfigurable processor in the second plurality ofreconfigurable processors;

the first plurality of buffers including a first set of sender buffersconfigured to receive data from the first reconfigurable processor andprovide the data to a second set of receiver buffers in the secondplurality of buffers, the second set of receiver buffers configured toprovide the data to the second reconfigurable processor;

the second plurality of buffers including a second set of sender buffersconfigured to receive data from the second reconfigurable processor andprovide the data to a first set of receiver buffers in the firstplurality of buffers, the first set of receiver buffers configured toprovide the data to the first reconfigurable processor; and

runtime logic, running on at least one reconfigurable processor in thefirst plurality of reconfigurable processors or the second ofreconfigurable processors, and configured to execute configuration filesthat define applications and application data for the applications usingthe first reconfigurable processor and the second reconfigurableprocessor, the execution including streaming configuration data (e.g.,bit stream) in the configuration files and the application data betweenthe first reconfigurable processor and the second reconfigurableprocessor using one or more buffers in the first plurality of buffersand one or more buffers in the second plurality of buffers.

12. A data processing system, comprising:

a plurality of reconfigurable processors including a firstreconfigurable processor and a second reconfigurable processor;

a first Smart Network Interface Controller (SmartNIC) operativelycoupled to the first reconfigurable processor, the first SmartNIC havinga first plurality of buffers;

a second SmartNIC operatively coupled to the second reconfigurableprocessor, the second SmartNIC having a second plurality of buffers; and

runtime logic, running on at least one reconfigurable processor in theplurality of reconfigurable processors, and configured to executeconfiguration files that define applications and application data forthe applications using the first reconfigurable processor and the secondreconfigurable processor, the execution including streamingconfiguration data (e.g., bit stream) in the configuration files and theapplication data between the first reconfigurable processor and thesecond reconfigurable processor using one or more buffers in the firstplurality of buffers and one or more buffers in the second plurality ofbuffers.

13. A data processing system, comprising:

a first reconfigurable processor having a first Network InterfaceController (NIC), and the first NIC having a first plurality of buffers;

a second reconfigurable processor having a second NIC, and the secondNIC having a second plurality of buffers; and

runtime logic, running on at least one reconfigurable processor (e.g.,the first reconfigurable processor, the second reconfigurable processor,a third reconfigurable processor), and configured to executeconfiguration files that define applications and application data forthe applications using the first reconfigurable processor and the secondreconfigurable processor, the execution including streamingconfiguration data (e.g., bit stream) in the configuration files and theapplication data between the first reconfigurable processor and thesecond reconfigurable processor using the first plurality of buffers ofthe first NIC and the second plurality of buffers of the second NIC.

14. A data processing system, comprising:

a first reconfigurable processor operatively coupled to a first NetworkInterface Controller (NIC);

a second reconfigurable processor operatively coupled to a second NIC;

an address generator of the first reconfigurable processor configured tostream configuration data (e.g., bit stream) and application data forexecution of configuration files from the first reconfigurable processorto the second reconfigurable processor using memory addresses that mapto a first plurality of buffers; and

an address generator of the second reconfigurable processor configuredto the configuration data (e.g., bit stream) and the application datafrom the second reconfigurable processor to the first reconfigurableprocessor using memory addresses that map to a second plurality ofbuffers.

15. A data processing system, comprising:

a pool of reconfigurable dataflow resources including a plurality ofprocessing nodes, respective processing nodes in the plurality ofprocessing nodes operatively coupled to respective pluralities ofreconfigurable processors and respective pluralities of buffers; and

a runtime processor operatively coupled to the pool of reconfigurabledataflow resources, and configured to:

-   -   receive an execution file for an application, the execution file        including configuration files for applications and        configurations of virtual dataflow resources required to execute        the configuration files, and the virtual dataflow resources        including a first virtual reconfigurable processor in a first        virtual processing node, a second virtual reconfigurable        processor in a second virtual processing node, and virtual        buffers that stream data between the first virtual        reconfigurable processor and the second virtual reconfigurable        processor;    -   allocate reconfigurable dataflow resources in the pool of        reconfigurable dataflow resources to the virtual dataflow        resources, the allocated reconfigurable dataflow resources        including        -   a first processing node in the respective processing nodes            allocated to the first virtual processing node,        -   a second processing node in the respective processing nodes            allocated to the second virtual processing node,        -   a first reconfigurable processor, operatively coupled to the            first processing node, allocated to the first virtual            reconfigurable processor,        -   a second reconfigurable processor operatively coupled to the            second processing node allocated to the second virtual            reconfigurable processor, and        -   a first plurality of buffers, operatively coupled to the            first processing node, and a second plurality of buffers,            operatively coupled to the second processing node, allocated            to the virtual buffers; and    -   execute the configuration files and process data for the        applications using the allocated reconfigurable dataflow        resources.

While the present invention is disclosed by reference to the preferredimplementations and examples detailed above, it is to be understood thatthese examples are intended in an illustrative rather than in a limitingsense. It is contemplated that modifications and combinations willreadily occur to those skilled in the art, which modifications andcombinations will be within the spirit of the invention and the scope ofthe following clauses.

What is claimed is:
 1. A data processing system, comprising: a pluralityof processing nodes, processing nodes in the plurality of processingnodes including a first processing node and a second processing node,the first processing node operatively coupled to the second processingnode, the first processing node having a first host processor, a firstplurality of reconfigurable processors operatively coupled to the firsthost processor, and a first plurality of Smart Network InterfaceControllers (SmartNICs) operatively coupled to the first plurality ofreconfigurable processors, and the second processing node having asecond host processor, a second plurality of reconfigurable processorsoperatively coupled to the second host processor, and a second pluralityof SmartNICs operatively coupled to the second plurality ofreconfigurable processors; a first plurality of buffers in a memory of afirst SmartNIC in the first plurality of SmartNICs, the first SmartNICoperatively coupled to a first reconfigurable processor in the firstplurality of reconfigurable processors; a second plurality of buffers ina memory of a second SmartNIC in the second plurality of SmartNICs, thesecond SmartNIC operatively coupled to a second reconfigurable processorin the second plurality of reconfigurable processors; the firstplurality of buffers including a first set of sender buffers configuredto receive data from the first reconfigurable processor and provide thedata to a second set of receiver buffers in the second plurality ofbuffers, the second set of receiver buffers configured to provide thedata to the second reconfigurable processor; the second plurality ofbuffers including a second set of sender buffers configured to receivedata from the second reconfigurable processor and provide the data to afirst set of receiver buffers in the first plurality of buffers, thefirst set of receiver buffers configured to provide the data to thefirst reconfigurable processor; and runtime logic configured to executeconfiguration files that define applications and process applicationdata for the applications using the first reconfigurable processor andthe second reconfigurable processor, the execution including streamingconfiguration data in the configuration files and the application databetween the first reconfigurable processor and the second reconfigurableprocessor using one or more buffers in the first plurality of buffersand one or more buffers in the second plurality of buffers, thereby thestreaming bypassing the first host processor and the second hostprocessor.
 2. The data processing system of claim 1, wherein theexecution includes streaming input data for the applications from thefirst reconfigurable processor to the second reconfigurable processor.3. The data processing system of claim 2, wherein one or more senderbuffers in the first set of sender buffers are configured to receive theinput data from the first reconfigurable processor and provide the inputdata to one or more receiver buffers in the second set of receiverbuffers, wherein the receiver buffers in the second set of receiverbuffers are configured to provide the input data to the secondreconfigurable processor.
 4. The data processing system of claim 3,wherein the execution includes streaming output data for theapplications from the second reconfigurable processor to the firstreconfigurable processor, wherein the output data is generated as aresult of processing the input data.
 5. The data processing system ofclaim 4, wherein one or more sender buffers in the second set of senderbuffers are configured to receive the output data from the secondreconfigurable processor and provide the output data to one or morereceiver buffers in the first set of receiver buffers, wherein thereceiver buffers in the first set of receiver buffers are configured toprovide the output data to the first reconfigurable processor.
 6. Thedata processing system of claim 5, wherein the first reconfigurableprocessor is configured to push the input data to the first SmartNIC,wherein the first SmartNIC is configured to write the input data intothe sender buffers in the first set of sender buffers, and wherein thefirst SmartNIC is configured to update tail pointers of the senderbuffers in the first set of sender buffers in response to the writing ofthe input data.
 7. The data processing system of claim 6, wherein thefirst SmartNIC is configured to send the input data to the secondSmartNIC in response to the updated tail pointers, wherein the secondSmartNIC is configured to write the input data into the receiver buffersin the second set of receiver buffers, and wherein the second SmartNICis configured to update tail pointers of the receiver buffers in thesecond set of receiver buffers in response to the writing of the inputdata.
 8. The data processing system of claim 7, wherein the secondreconfigurable processor is configured to pull the input data from thesecond SmartNIC by reading the input data from the receiver buffers inthe second set of receiver buffers in response to the updated tailpointers.
 9. The data processing system of claim 8, wherein the secondreconfigurable processor is configured to push the output data to thesecond SmartNIC, wherein the second SmartNIC is configured to write theoutput data into the sender buffers in the second set of sender buffers,and wherein the second SmartNIC is configured to update tail pointers ofthe sender buffers in the second set of sender buffers in response tothe writing of the output data.
 10. The data processing system of claim9, wherein the second SmartNIC is configured to send the output data tothe first SmartNIC in response to the updated tail pointers, wherein thefirst SmartNIC is configured to write the output data into the receiverbuffers in the first set of receiver buffers, and wherein the firstSmartNIC is configured to update tail pointers of the receiver buffersin the first set of receiver buffers in response to the writing of theoutput data.
 11. The data processing system of claim 10, wherein thefirst reconfigurable processor is configured to pull the output datafrom the first SmartNIC by reading the output data from the receiverbuffers in the first set of receiver buffers in response to the updatedtail pointers.
 12. The data processing system of claim 1, wherein anetwork fabric operatively couples the first processing node and thesecond processing node, and the network fabric is configured toimplement the streaming of the input data from the sender buffers in thefirst set of sender buffers to the receiver buffers in the second set ofreceiver buffers, and to implement the streaming of the output data fromthe sender buffers in the second set of sender buffers to the receiverbuffers in the first set of receiver buffers.
 13. The data processingsystem of claim 1, wherein the first reconfigurable processor notifiesthe second reconfigurable processor of remote invocations using one ormore remote procedure calls.
 14. The data processing system of claim 13,wherein the first reconfigurable processor uses the sender buffers inthe first set of sender buffers and the receiver buffers in the secondset of receiver buffers to send one or more argument values to thesecond reconfigurable processor for execution of the remote procedurecalls.
 15. The data processing system of claim 1, wherein the secondreconfigurable processor notifies the first reconfigurable processor ofremote invocations using one or more remote procedure calls.
 16. Thedata processing system of claim 15, wherein the second reconfigurableprocessor uses the sender buffers in the second set of sender buffersand the receiver buffers in the first set of receiver buffers to sendone or more argument values to the first reconfigurable processor forexecution of the remote procedure calls.
 17. The data processing systemof claim 1, wherein respective SmartNICs in the first plurality ofSmartNICs are operatively coupled to respective reconfigurableprocessors in the first plurality of reconfigurable processors byrespective buses, and respective SmartNICs in the second plurality ofSmartNICs are operatively coupled to respective reconfigurableprocessors in the second plurality of reconfigurable processors byrespective buses.
 18. The data processing system of claim 1, wherein theconfiguration files include a plurality of functions.
 19. The dataprocessing system of claim 1, further comprising: the runtime logicconfigured to execute a first set of functions in the plurality offunctions on the first reconfigurable processor, and a second set offunctions in the plurality of functions on the second reconfigurableprocessor, wherein functions in the second set of functions and/or datatherefor are transmitted to the second reconfigurable processor usingthe sender buffers in the first set of sender buffers and the receiverbuffers in the second set of receiver buffers, and wherein results ofexecuting the functions and/or the data therefor on the secondreconfigurable processor are transmitted to the first reconfigurableprocessor using the sender buffers in the second set of sender buffersand the receiver buffers in the first set of receiver buffers.
 20. Thedata processing system of claim 1, wherein the buffers in the firstplurality of buffers and the buffers in the second plurality of buffersare First-In, First-Out (FIFO) buffers.
 21. The data processing systemof claim 1, wherein the runtime logic runs in at least one of the firsthost processor and the second host processor.